Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Triage 16 reference-suspect cases from the Claude 5 failure audit

Aperta
  • Qualcosa non va?
#91 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
data, testing-qa

Direzione di ricerca

Inizia con i 16 casi del manifest congelato in paper/snapshot/20260501/us_scenarios.csv e confronta i relativi valori in us_reference_outputs.csv con le derivazioni dichiarate di PolicyEngine 4.16.1 e policyengine-us 1.723.0. Verifica ogni caso rispetto a policyengine-us, in particolare le regole relative a CHIP, Medicaid, Head Start, SNAP e alle imposte statali. Il lavoro è completato quando ogni caso sospetto ha un errore di riferimento confermato, un problema o una correzione upstream identificati, oppure un motivo documentato per cui il riferimento è corretto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

The Codex failure-audit of the Claude 5 models' 322 wrong cases (PR #90) flags 16 cases where the judge suspects the PolicyEngine reference value rather than the models — clustering in CHIP/Medicaid income-threshold handling, three state income tax cells, two SNAP cells, and Head Start eligibility. Each needs human triage against policyengine-us; the June audit's equivalents produced real upstream fixes.

The verdicts are conservative by prompt design (default is models-wrong), so these are worth taking seriously. Reproduction: every case is a frozen-manifest cell — scenario inputs in paper/snapshot/20260501/us_scenarios.csv, reference values in us_reference_outputs.csv (policyengine.py 4.16.1 / policyengine-us 1.723.0).

scenario_032 · child1_chip_eligible

PolicyEngine's CHIP reference appears to use an implausibly low Minnesota child CHIP income threshold, treating 275% FPL for a family of three as about $38,000 when it should be far higher, making the stated $42,664 income likely below rather than above the threshold.

scenario_026 · child1_medicaid_eligible

The recorded reference value appears inverted or mismatched: it says child1_medicaid_eligible = Yes while the supplied PolicyEngine derivation explicitly says PolicyEngine determined Child 1 is not Medicaid eligible (False).

scenario_064 · child1_medicaid_eligible

The recorded reference value is No, but the provided PolicyEngine derivation explicitly says PolicyEngine determined child1_medicaid_eligible is True, suggesting a reference extraction or label mismatch for this variable.

scenario_038 · child2_chip_eligible

PolicyEngine appears to apply an implausibly low Louisiana CHIP income threshold, treating 200% FPL for a four-person household as about $21,600 when 200% FPL should be far higher than this household's roughly $22,992 income.

scenario_100 · child2_head_start_eligible

PolicyEngine appears to be denying Head Start eligibility despite a 5-year-old child in a three-person household with $5,915 annual wages, which is plainly below the federal poverty guideline threshold described in the reference rationale.

scenario_008 · child3_chip_eligible

The derivation appears to apply the CHIP income threshold backwards or uses an incorrect New Jersey 2026 threshold, since an 8-person household with about $30,915 income is far below any plausible child CHIP/NJ FamilyCare income limit expressed as a multiple of FPL.

scenario_109 · child3_chip_eligible

PolicyEngine appears to be using an implausibly low 200% FPL threshold for a five-person household in 2026, causing Florida CHIP eligibility to be denied despite $30,000 being below the likely applicable income limit.

scenario_026 · child3_medicaid_eligible

PolicyEngine appears to be returning Medicaid eligibility for a 9-year-old NC child despite household income exceeding the stated child Medicaid/CHIP income threshold, possibly conflating CHIP eligibility with Medicaid eligibility or applying the wrong income limit.

scenario_008 · child5_head_start_eligible

PolicyEngine appears to compare the household's roughly $30,915 income against an incorrect Head Start poverty threshold for an 8-person household, since that income is well below the federal poverty guideline rather than above it.

scenario_015 · head_medicaid_eligible

PolicyEngine appears to have evaluated the head under Indiana's non-disabled adult Medicaid pathway despite the prompt setting is disabled=true, potentially missing a disability or working-disabled Medicaid category.

scenario_031 · head_medicaid_eligible

PolicyEngine appears to have applied California's MAGI Medicaid expansion logic for non-elderly adults despite the head being age 67, where aged Medicaid rules should govern.

scenario_038 · snap

The stated PolicyEngine derivation says the monthly SNAP allotment is $994 minus a $392.70 expected contribution, or $601.30, but $601.30 times 12 is $7,215.60 rather than the reported $7,286.94.

scenario_121 · snap

PolicyEngine appears to be applying the SNAP gross income limit to a household with a disabled member, even though elderly/disabled SNAP households are generally subject to the net income test rather than the gross income test, and the stated medical expenses could reduce net income enough to produce a positive benefit.

scenario_046 · state_income_tax_before_refundable_credits

PolicyEngine's derivation says Oklahoma itemized deductions of $22,429.78 exceeded the $12,700 standard deduction, but the taxable income of $75,303 appears to subtract the standard deduction rather than the larger itemized deduction.

scenario_056 · state_income_tax_before_refundable_credits

PolicyEngine appears to apply New Jersey's rate schedule to taxable income after the $1,000 exemption while omitting New Jersey's low-income filing/taxability threshold that eliminates liability for a single filer with gross income under $10,000.

scenario_085 · state_income_tax_before_refundable_credits

PolicyEngine appears to have incorrectly zeroed Pennsylvania income tax before refundable credits, likely by applying an erroneous income threshold/tax-forgiveness rule or excluding taxable interest income that Pennsylvania taxes at its flat rate.

🤖 Generated with Claude Code

Lingua principale
Python
Stelle
2
Fork
3
Merge medio
11h 11m
PR unite (30g)
22

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di PolicyEngine/policybench

Tutte le issue di PolicyEngine/policybench

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.