Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Stamp US year files with the register of renamed stored inputs they were cut under

Open
  • Is something wrong?
#532 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data

Research direction

Trace create_datasets(), ensure_datasets(), and load_datasets(), then inspect policyengine_legacy_input_renames and the RENAMES_H5_DATASET handling described in #531. Implement the year-file stamp and validation paths, preserving the stated error propagation behavior. Done means files with mismatched or invalid metadata are regenerated or refused as specified, without inspecting tables or source data.

Written by the indexing model from the issue text.

Description

Follow-up to #530 / #531.

Problem

ensure_datasets() reuses every US year file ({data_folder}/{stem}_year_{year}.h5) that already exists and passes its staleness check. load_datasets() loads them. A year file that lost a renamed stored input gives wrong results without any error:

  • policyengine.py 6.1.1 and earlier. create_datasets() exported only sim.input_variables. policyengine-us 2.x renamed would_claim_wic to takes_up_wic_if_eligible, so these files store neither name. Every WIC-eligible person then takes WIC up: about $11.5B instead of $6.6B in 2024 (PolicyEngine/microcosm#1026).
  • #531 (unreleased draft). Year files record the renames that were applied (policyengine_legacy_input_renames). ensure_datasets() regenerates, and load_datasets() refuses, a file with no record. The record shows that a file was cut with the mapping, not under which register. #531's own body and the comment on RENAMES_H5_DATASET say so. When policyengine-us next renames an input and LEGACY_INPUT_RENAMES gains an entry, year files cut before that entry will still carry a record and be reused, and they can lose the newly renamed input the same way.

A subset check ("the file's register covers today's") would not close this either. A file cut by a newer release under a larger register stores the new live name. An older release whose engine does not define that name would skip it (_build_simulation_from_dataset sets only columns in system.variables). A removed or retargeted entry fails the same way.

Proposal

  • create_datasets() stamps each year file last with policyengine_year_file_stamp, stored as UTF-8 JSON {"register": <the whole LEGACY_INPUT_RENAMES>, "revision": <year-file format revision>}.
  • ensure_datasets() regenerates, and load_datasets() refuses, any year file whose stamp does not equal this release's stamp exactly, or whose renames record is missing, malformed or not covered by its stamped register. The error names the renames the file may lack, generated from the register with no WIC special case.
  • The check reads only the stamp and the record, never the tables or the source dataset.
  • Permission, directory and file-lock errors propagate instead of starting a full regeneration.

Saved simulation outputs (Simulation.load()) still gate only on the record's presence. That is left to a separate issue.

Dominant language
Python
Stars
7
Forks
9
Avg merge
17h 40m
Merged PRs (30d)
10

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PolicyEngine/policyengine.py

All issues in PolicyEngine/policyengine.py

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.