Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

Register weighted-microdata rules: weights aren't values

Geschlossen
  • Stimmt etwas nicht?
#806 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Ein zugehöriger Pull Request wurde bereits gemerged.

  • #807 von @anth-volk — gemerged

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
30/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
pandas, python
Bereich
data, testing

Rechercherichtung

Start with rules/registry.toml to see how existing rules and their AST guards and proofs are declared, then read simulation_output_common.py, where raw values are read from weighted frames. The output-builder tests that currently use plain pd.DataFrames are the other place to look. Done means the four named rules are registered with guards and proofs, the confined-read exception list is in place, and the builder tests use one shared weighted factory.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Problem

The executor's entity frames are microdf MicroDataFrames weighted by <entity>_weight, so .sum(), .mean() and .count() on any column are already weighted.

In #766 (fixed by #767), build_budgetary_impact passed "household_weight" to _sum_output_variable, the helper it uses for ordinary values. That weighted the weight column by itself, giving the sum of squared weights.

  • Effect: budget.households was about 10^4 times too high on every run (New York 2026: 135.6 billion, against about 7.5 million households). Clients divide by it, so the average change per household was about 10^4 times too small.
  • Why it escaped: the builder test used plain pd.DataFrames, which carry no weights.

Nothing stops the same class of mistake from coming back.

Proposal

Phase 1 step 1.9. Register the rule that weights aren't values in rules/registry.toml, with AST guards and proofs:

  • sim.microdata.weights-not-values (doc-only): weights, values, labels and counts each allow different operations.
  • sim.microdata.weights-not-aggregated-as-values: a weight is never the variable of a weighted aggregator, and never frame[<weight>] followed by a weighted reduction.
  • sim.microdata.weight-stripping-confined: raw values are read from a weighted frame only in simulation_output_common.py.
    • Counting entities is always allowed there.
    • Any other read there is a recorded exception, in a list that may only shrink.
  • sim.microdata.builder-tests-use-weighted-frames: output-builder tests build entity frames through one shared weighted factory, never as plain pd.DataFrames.

Alongside, two cheap runtime checks that the AST guards can't replace:

  • change totals (reform minus baseline) log a warning when the two sides carry different weights;
  • an implausible published household count fails the run.

Decisions

  • UK local-authority handoff. The breakdown hands policyengine's longwise geography impact a plain household frame, and that function weights by household_weight itself. The handoff moves into simulation_output_common.py as the stripping rule's one recorded exception. Removing it is tracked separately.
  • Weight mismatch: warn first. It logs a warning and still publishes the total. It becomes an error once logs show that no legitimate run produces a mismatch.
  • Household count: fail the run. A failed run is better than a headline that's 10^4 times wrong.
Vorherrschende Sprache
Python
Sterne
1
Forks
1
Ø Merge
1 T. 19 Std.
Gemergte PRs (30 T.)
77

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus PolicyEngine/policyengine-sim-api

Alle Issues in PolicyEngine/policyengine-sim-api

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.