Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Unknown-column warning hides removed model inputs behind raw source columns (ten-name cap)

Open
  • Is something wrong?
#609 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start by reading the aggregated unknown-column warning implementation and the #531 tests in policyengine-core; the issue does not name their file paths. Compare the warning behavior with policyengine-us test tests/microsimulation/data/test_removed_mortgage_interest_inputs.py and its expected removed input names. Done means a dataset with more than ten unknown columns triggers a warning that names the registered removed input and its guidance, while the existing #531 tests still pass.

Written by the indexing model from the issue text.

Description

Follow-up to #530 / #531 (the aggregated unknown-column warning), found while removing two inputs from policyengine-us (PolicyEngine/policyengine-us#9605).

Problem

The warning names only the first ten unknown columns, sorted, plus a count. On a dataset that also carries raw source columns, those ten slots are taken by names that were never model variables, so a column for a removed model input is never named.

Measured 2026-09-25 by loading the certified US default dataset (populace_us_2024.h5@populace-us-2024-spm-20260909, sha256 6496cc43…) through policyengine_us.Microsimulation on the #9605 branch with policyengine-core 3.32.7. The single warning was:

The dataset contains 191 column(s) that do not match any variable in the tax-benefit system and were ignored: ACTC_CRD, AGI, ANN_VAL, A_AGE, A_ENRLW, A_EXPRRP, A_FAMREL, A_FAMTYP, A_FNLWGT, A_FTPT, … (181 more)

That branch deletes first_home_mortgage_interest and second_home_mortgage_interest, and the dataset carries both, but neither appears: uppercase survey columns sort first. The warning fires on this file whether or not a model input was removed, so it carries no signal there. On a dataset with no raw source columns it works as intended (policyengine-us test tests/microsimulation/data/test_removed_mortgage_interest_inputs.py asserts both names appear).

Options

  1. Removed-input registry (preferred). Let a tax-benefit system declare removed or renamed inputs (name → replacement or guidance). At dataset load, emit a separate warning naming every dataset column that matches the registry, independent of the ten-name cap. This is the only option that tells a user what to do.
  2. Full list at a lower level. Keep the capped WARNING and also log the complete sorted list at INFO or DEBUG.
  3. Ordering. List snake_case names (the model's convention) before other names, so removed inputs outrank raw survey columns. Cheap, but a heuristic.

Acceptance

  • A test that builds a dataset with more than ten unknown columns, one of them a registered removed input, and asserts the warning output names that input and its guidance.
  • The existing #531 tests still pass (known columns load unchanged; clean datasets stay quiet).

The data-side counterpart (certified files not shipping raw source columns or stale inputs) is tracked in PolicyEngine/microcosm#673.

Dominant language
Python
Stars
22
Forks
30
Avg merge
2d 19h
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PolicyEngine/policyengine-core

All issues in PolicyEngine/policyengine-core

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.