Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Bake per-state ACS-local year files into Stage 12 and record the dataset state runs load

Aperta
  • Qualcosa non va?
#884 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, data-engineering

Direzione di ricerca

Read #878 and inspect Stage 12's precompute, cache manifest, image fetch, worker check, runtime lookup, and stage12_adapter.py, along with the shared simulation_runtime helpers. Trace which dataset each region type loads and how its provenance and revision reach run records; done means Stage 12 has per-state files, validates loaded dataset identity, preserves compatibility during rollout, and reports the loaded revision as data_version.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

#878 makes the main simulation path pre-build one ACS-local year file per state, so state and district requests stop preparing the 9.8 GB regional dataset on each new worker. Stage 12 has the same problem and needs its own copy of that change.

  • Slow loading. Stage 12's US workers resolve state and district requests to populace_us_2024_acs_local through the shared simulation_runtime helpers. Each new worker downloads the full source and prepares the full-country year file. On 21 Oct, Utah's dataset_load took 55 s on a reused worker and 250 s on a new one, and the workers reached about 51 GB of memory.

Stage 12 also records the wrong dataset for these runs:

  • Wrong dataset recorded. The entry adapter (stage12_adapter.py) records the national dataset populace_us_2024 as the dataset identity, URI and population for every US request. State and district runs actually load populace_us_2024_acs_local, so their comparison records name the wrong dataset.
  • Worker checks. _require_installed_bundle requires the default dataset, and nothing checks that the dataset recorded is the one loaded.
  • Wrong data_version on the main path. For state runs, the main path names the national release (populace-us-2024-spm-20260915) as data_version, because nothing passes the loaded dataset's revision.

Change

  1. Stage 12 state files. Stage 12's precompute, cache manifest, image fetch, worker check and runtime lookup get their own copy of #878's per-state datasets. The two stacks stay separate.
  2. Published bundle. It records which dataset each region type loads. The v2 version manifest goes under a new key that only updated entry services read. The existing keys keep their shapes for entry services already running.
  3. Workers. They accept regional provenance and check that it names the dataset actually loaded. During the rollout, the default dataset stays accepted for state and district runs from entry services that haven't been updated.
  4. Entry adapter. It sends the regional dataset's provenance for state and district requests.
  5. Main path. State and district runs report the loaded dataset's revision as data_version, so both paths report the same value.

Related: #878, #717.

Lingua principale
Python
Stelle
1
Fork
1
Merge medio
1g 16h
PR unite (30g)
60

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di PolicyEngine/policyengine-sim-api

Tutte le issue di PolicyEngine/policyengine-sim-api

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.