Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Bake per-state ACS-local year files into Stage 12 and record the dataset state runs load

Aberta
  • Algo errado?
#884 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
backend, data-engineering

Direção de pesquisa

Read #878 and inspect Stage 12's precompute, cache manifest, image fetch, worker check, runtime lookup, and stage12_adapter.py, along with the shared simulation_runtime helpers. Trace which dataset each region type loads and how its provenance and revision reach run records; done means Stage 12 has per-state files, validates loaded dataset identity, preserves compatibility during rollout, and reports the loaded revision as data_version.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

#878 makes the main simulation path pre-build one ACS-local year file per state, so state and district requests stop preparing the 9.8 GB regional dataset on each new worker. Stage 12 has the same problem and needs its own copy of that change.

  • Slow loading. Stage 12's US workers resolve state and district requests to populace_us_2024_acs_local through the shared simulation_runtime helpers. Each new worker downloads the full source and prepares the full-country year file. On 21 Oct, Utah's dataset_load took 55 s on a reused worker and 250 s on a new one, and the workers reached about 51 GB of memory.

Stage 12 also records the wrong dataset for these runs:

  • Wrong dataset recorded. The entry adapter (stage12_adapter.py) records the national dataset populace_us_2024 as the dataset identity, URI and population for every US request. State and district runs actually load populace_us_2024_acs_local, so their comparison records name the wrong dataset.
  • Worker checks. _require_installed_bundle requires the default dataset, and nothing checks that the dataset recorded is the one loaded.
  • Wrong data_version on the main path. For state runs, the main path names the national release (populace-us-2024-spm-20260915) as data_version, because nothing passes the loaded dataset's revision.

Change

  1. Stage 12 state files. Stage 12's precompute, cache manifest, image fetch, worker check and runtime lookup get their own copy of #878's per-state datasets. The two stacks stay separate.
  2. Published bundle. It records which dataset each region type loads. The v2 version manifest goes under a new key that only updated entry services read. The existing keys keep their shapes for entry services already running.
  3. Workers. They accept regional provenance and check that it names the dataset actually loaded. During the rollout, the default dataset stays accepted for state and district runs from entry services that haven't been updated.
  4. Entry adapter. It sends the regional dataset's provenance for state and district requests.
  5. Main path. State and district runs report the loaded dataset's revision as data_version, so both paths report the same value.

Related: #878, #717.

Linguagem predominante
Python
Estrelas
1
Forks
1
Merge médio
1d 16h
PRs com merge (30d)
60

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de PolicyEngine/policyengine-sim-api

Todas as issues de PolicyEngine/policyengine-sim-api

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.