Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Register weighted-microdata rules: weights aren't values

Aberta
  • Algo errado?
#806 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@anth-volk já está trabalhando nisso.

Desde 14/10/2026.

  • #807 de @anth-volk — aberto

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
30/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
pandas, python
Domínio
data, testing

Direção de pesquisa

Start with rules/registry.toml to see how existing rules and their AST guards and proofs are declared, then read simulation_output_common.py, where raw values are read from weighted frames. The output-builder tests that currently use plain pd.DataFrames are the other place to look. Done means the four named rules are registered with guards and proofs, the confined-read exception list is in place, and the builder tests use one shared weighted factory.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Problem

The executor's entity frames are microdf MicroDataFrames weighted by <entity>_weight, so .sum(), .mean() and .count() on any column are already weighted.

In #766 (fixed by #767), build_budgetary_impact passed "household_weight" to _sum_output_variable, the helper it uses for ordinary values. That weighted the weight column by itself, giving the sum of squared weights.

  • Effect: budget.households was about 10^4 times too high on every run (New York 2026: 135.6 billion, against about 7.5 million households). Clients divide by it, so the average change per household was about 10^4 times too small.
  • Why it escaped: the builder test used plain pd.DataFrames, which carry no weights.

Nothing stops the same class of mistake from coming back.

Proposal

Phase 1 step 1.9. Register the rule that weights aren't values in rules/registry.toml, with AST guards and proofs:

  • sim.microdata.weights-not-values (doc-only): weights, values, labels and counts each allow different operations.
  • sim.microdata.weights-not-aggregated-as-values: a weight is never the variable of a weighted aggregator, and never frame[<weight>] followed by a weighted reduction.
  • sim.microdata.weight-stripping-confined: raw values are read from a weighted frame only in simulation_output_common.py.
    • Counting entities is always allowed there.
    • Any other read there is a recorded exception, in a list that may only shrink.
  • sim.microdata.builder-tests-use-weighted-frames: output-builder tests build entity frames through one shared weighted factory, never as plain pd.DataFrames.

Alongside, two cheap runtime checks that the AST guards can't replace:

  • change totals (reform minus baseline) log a warning when the two sides carry different weights;
  • an implausible published household count fails the run.

Decisions

  • UK local-authority handoff. The breakdown hands policyengine's longwise geography impact a plain household frame, and that function weights by household_weight itself. The handoff moves into simulation_output_common.py as the stripping rule's one recorded exception. Removing it is tracked separately.
  • Weight mismatch: warn first. It logs a warning and still publishes the total. It becomes an error once logs show that no legitimate run produces a mismatch.
  • Household count: fail the run. A failed run is better than a headline that's 10^4 times wrong.
Linguagem predominante
Python
Estrelas
1
Forks
1
Merge médio
1d 11h
PRs com merge (30d)
50

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de PolicyEngine/policyengine-sim-api

Todas as issues de PolicyEngine/policyengine-sim-api

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.