Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Bake per-state ACS-local year files into Stage 12 and record the dataset state runs load

Abierto
  • ¿Algo no está bien?
#884 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, data-engineering

Línea de trabajo

Read #878 and inspect Stage 12's precompute, cache manifest, image fetch, worker check, runtime lookup, and stage12_adapter.py, along with the shared simulation_runtime helpers. Trace which dataset each region type loads and how its provenance and revision reach run records; done means Stage 12 has per-state files, validates loaded dataset identity, preserves compatibility during rollout, and reports the loaded revision as data_version.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

#878 makes the main simulation path pre-build one ACS-local year file per state, so state and district requests stop preparing the 9.8 GB regional dataset on each new worker. Stage 12 has the same problem and needs its own copy of that change.

  • Slow loading. Stage 12's US workers resolve state and district requests to populace_us_2024_acs_local through the shared simulation_runtime helpers. Each new worker downloads the full source and prepares the full-country year file. On 21 Oct, Utah's dataset_load took 55 s on a reused worker and 250 s on a new one, and the workers reached about 51 GB of memory.

Stage 12 also records the wrong dataset for these runs:

  • Wrong dataset recorded. The entry adapter (stage12_adapter.py) records the national dataset populace_us_2024 as the dataset identity, URI and population for every US request. State and district runs actually load populace_us_2024_acs_local, so their comparison records name the wrong dataset.
  • Worker checks. _require_installed_bundle requires the default dataset, and nothing checks that the dataset recorded is the one loaded.
  • Wrong data_version on the main path. For state runs, the main path names the national release (populace-us-2024-spm-20260915) as data_version, because nothing passes the loaded dataset's revision.

Change

  1. Stage 12 state files. Stage 12's precompute, cache manifest, image fetch, worker check and runtime lookup get their own copy of #878's per-state datasets. The two stacks stay separate.
  2. Published bundle. It records which dataset each region type loads. The v2 version manifest goes under a new key that only updated entry services read. The existing keys keep their shapes for entry services already running.
  3. Workers. They accept regional provenance and check that it names the dataset actually loaded. During the rollout, the default dataset stays accepted for state and district runs from entry services that haven't been updated.
  4. Entry adapter. It sends the regional dataset's provenance for state and district requests.
  5. Main path. State and district runs report the loaded dataset's revision as data_version, so both paths report the same value.

Related: #878, #717.

Lenguaje dominante
Python
Estrellas
1
Forks
1
Merge medio
1 d 16 h
PR fusionados (30 d)
60

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de PolicyEngine/policyengine-sim-api

Todos los issues de PolicyEngine/policyengine-sim-api

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.