Load the ACS local-area dataset for US state and district runs

Offen
  • Stimmt etwas nicht?
#717 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Dieses Issue hat noch niemand übernommen.

  • #726 von @anth-volk — ohne Merge geschlossen

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
backend, data, performance

Rechercherichtung

Start with _region_parent_dataset_reference and _resolve_region in projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_runtime.py, then inspect runtime_dataset_uri(...) and the linked build_populace_state_slices.py approach. Done means state and district runs use validated local-area extracts with linked groups and unchanged weights, while the resolved URI preserves the overlay revision and sha256.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Problem

Every US state/xx and congressional-district run executes on the national default file. In projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_runtime.py:

  • _resolve_region (lines 317–360) uses region.dataset_path when it is set.
  • Otherwise _region_parent_dataset_reference (lines 276–301) walks up to the national region and returns its dataset.

policyengine.py state and district regions carry no dataset_path, so the fallback always applies (PolicyEngine/policyengine.py#551).

What changes once regions carry the local-area dataset

PolicyEngine/policyengine.py#551 proposes resolving state and district regions to populace_us_2024_acs_local.h5. The executor would then receive that file for every state run:

  • Size: 9.8 GB. It covers 1,588,854 households (Sep 23 build 767312d60).
  • Memory: our Modal jobs needed a 64 GB container to load it and extract one state.
  • Per-state counts: Pennsylvania alone is 64,981 households. Child-poverty-impact-dashboard slices range from 3.4k households (WY) to 163k (CA).

Suggested direction

  • Build per-state extracts once per local-area release, keeping every linked person, tax unit, SPM unit, family and marital unit, with weights unchanged. Run state and district requests on the extract.
  • PolicyEngine/child-poverty-impact-dashboard scripts/build_populace_state_slices.py does this for 51 states. It checks that slices partition the frame exactly and that no group straddles a state boundary. It also verified a 2021-CTC reform on all 51 slices against masked full-file runs (population and transfer total within 1e-6, poverty rates within 1e-4).
  • Confirm the resolved URI keeps the overlay's own revision and sha256. Today runtime_dataset_uri(...) passes bundle.data_artifact_revision (the national artifact's revision) as the artifact revision, so check that it does not override the local-area pin.

Related

  • Region wiring: PolicyEngine/policyengine.py#551
  • API v1 state reports: PolicyEngine/policyengine-api#3864
  • Weight concentration on the local-area build: PolicyEngine/microcosm#403

🤖 Generated with Claude Code

Vorherrschende Sprache
Python
Sterne
1
Forks
1
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
83

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus PolicyEngine/policyengine-sim-api

Alle Issues in PolicyEngine/policyengine-sim-api

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.