Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

The PUF donor's amounts stop at TY2021 levels, two columns at TY2015, and nothing ages them to the build year

Open
#1,248 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

Start with tools/build_us_puf_support_base.py::_puf_tax_unit_donor_from_h5, then trace puf_tax_unit_donor_from_arrays and the uprate source spec in us/spec/sources.yaml; compare donor columns with the archived uprate_puf method and the receipts in receipts/issue_tables.md. Resolve the listed series, synthetic dividend, donor-to-arm, target, and #1191 questions before defining a build comparison. Done means an agreed rebase that ages each supported donor series to the build year without applying the 2015–2021 leg twice.

Written by the indexing model from the issue text.

Description

Summary

The processed PUF donor Microcosm US reads (puf_2024.h5, sha256 7669f5b5…d6df) is a TY2021 file. Checked record by record against the TY2015 source it was built from:

  • 34 directly mapped dollar columns equal the archived generator's 2015-to-2021 value on 100% of their nonzero source records, and none matches a 2022 or 2023 value.
  • Two columns carry no uprating at all: partnership income and farm operations income are at TY2015 levels.
  • Schedule C income carries two factors, and the interest-paid columns match on 76.1% of records; a separate structural conversion moved the rest.

The donor's weighted totals sit on the SOI TY2021 actuals: $9,078B of wages against $9,022B for TY2021 and $10,204B for TY2023.

Microcosm ages none of it. The source spec declares an uprate from 2015 to the build year for the PUF stage. Nothing runs it, and run as written on this donor it would count 2015-to-2021 growth twice.

The leg that never ran is large and changes sign by series, because TY2021 was a pandemic and market-peak year. Per return, from TY2021 to TY2023 by the archived method: wages ×1.133, taxable interest ×3.035, ordinary dividends ×1.305, Social Security ×1.232, capital gain distributions ×0.392.

The 30 September Route A release's calibration residuals do not show this cost on their own. After the release's value repairs and before calibration, the frame's PUF arm already carries more than the donor does. Its tax-unit base is 1.14 times the donor's, and it carries 1.09 times the donor's wages, 1.50 times its taxable interest, 1.83 times its taxable pensions and 2.00 times its long-term gains. That second effect is as large as the missing leg and mostly runs the other way. Taxable interest ends 16% short of its national target today. By arithmetic on the design-weight parts, aging that donor column alone would start it about 35% over.

Proposal. Nothing is changed here. Rebase the donor in Microcosm: verify the factor each column carries against the TY2015 source the build already reads, then replace it with that column's own series from TY2015 to the build year. The step that carries donor values onto the PUF arm needs its own look before or with it.

This also corrects statements already on the tracker and in the code:

  • #487 describes the donor as "uprated to 2024". Its 2.39× on capital gain distributions compares the donor's $24.3B with the target, the TY2023 total of $9.3B aged to $10.16B for 2024. The donor sits on the SOI TY2021 total of $23.9B, so the 2.39× is the distance from TY2021 to that target.
  • The 2026-09-22 comment on #981 says the 2021-to-2024 leg multiplies each variable, that "the PUF is frozen at 2024", and that the build consumes the 1.8.0 release file as pinned. The donor is frozen at TY2021, and it is a later, larger file than the 1.8.0 asset.

Rulings asked

  1. Where the fix goes: rebase in Microcosm's donor path, as proposed, or one of the three alternatives under "Proposal".
  2. The open series choices in the proposal's table.
  3. Synthetic dividends. The 3,985 records that replace the PUF's four aggregate records have no source record to recover TY2015 amounts from. Proposed: qualified × growth(qualified, 2015, Y) / 1.3628, and non-qualified × growth(non-qualified, 2015, Y) / 1.3914. The second denominator, the ordinary-dividend factor, is a convention, not a verified carried factor: non-qualified dividends have none. Both keep signs. Also needed: whether to clip the 1,190 negative non-qualified amounts these records already hold (−$1.42B at the donor's weights, against $25.6B of qualified dividends on the same records).
  4. The donor-to-arm step. Whether it is examined before the rebase or in the same build, and who owns it. Proposed: a follow-up issue. By the arithmetic below, a rebase alone overshoots taxable interest.
  5. Conflicting national targets. Which row stands for ordinary dividends ($457.4B from TY2022 Historic Table 2, all returns; $544.8B from TY2023 Table 4.3, returns excluding dependents) and for taxable Social Security ($510.8B TY2022-based, $573.0B TY2023-based). The rows differ in vintage and, for dividends, in return population, yet the build compiles each pair to one identical household column, so the dependents exclusion is not applied. Any before-and-after comparison of a rebase depends on the answer; a follow-up issue could own it.
  6. With #1191. One build with #1191's rebind and gross-up, or the rebase first.

What the donor carries

Paths below are under packages/microcosm-build/src/microcosm/build/ unless they start with tools/.

The published release. All 74 arrays of puf_2021.h5 and puf_2024.h5 in irs-soi-puf 1.8.0 are identical.

The build donor against its source. The donor's 207,692 regular tax units are the source's records in the same order. The archived uprate_puf (policyengine-us-data 42ed5d45, datasets/puf/uprate_puf.py), run unchanged on the source for 2021, 2022 and 2023, gives four hypotheses per field with the raw file. Each donor column was compared with its archived source expression under each.

What the column carries over its TY2015 fields Columns Result
Its own SOI line, per return, to TY2021 9: wages ×1.1871, taxable interest ×1.0105, tax-exempt interest ×0.8397, qualified dividends ×1.3628, non-qualified dividends (ordinary ×1.3914 less qualified), capital gain distributions ×1.9333, taxable IRA distributions ×1.5092, taxable pensions ×1.1637, Social Security ×1.2234 100% of nonzero source records on every column; 0% at 2015, 2022 or 2023
A flat 2% a year to 2021, ×1.1262 4: cash charitable gifts, real estate taxes, and E19200 interest, which Microcosm splits into mortgage and investment interest 100% on the first two. Interest: 76.1%; of the other 21,765 records, 20,522 sit above the value and 1,243 are zero
AGI per return to TY2021, ×1.3560 23, including both Schedule D gains, rental income, estate income and miscellaneous income 100% on every column
Its own sign-split line, then AGI growth again 2: Schedule C income and its share from a specified service trade or business (SSTB) (×1.6739 on profits) 100%
Nothing: TY2015 levels 2: partnership_income (E26190 − E26180 + E25980 − E25960) and farm_operations_income (E02100) 100% equal the raw fields
Derived or simulated from the amounts above 11: qualified business income (QBI) W-2 wages and property basis with their SSTB shares; real estate investment trust, publicly traded partnership and business development company income; partnership self-employment earnings; structural mortgage balances and interest not testable record by record
The TY2015 source value, by construction 1: puf_source_adjusted_gross_income equals E00100 on every regular record
Zero by construction 4: s_corp_income and three Social Security components all zero
Design weight 1: S006 / 100 × 1.0686, return-count growth to 2021 every regular record
Not a dollar amount 25: identifiers, demographics, predictors, flags, origination years

That is all 82 columns of the donor table Microcosm's own loader builds from these files (puf_donor_columns in the base summary). The per-column table is in receipts/issue_tables.md.

Four details:

  • The two unuprated columns read fields that none of the archived generator's lists name. In the published 1.8.0 file the same array was E26270 aged to 2021; the generator has built it from the split components since 2025-05-22 (c33e3bff).
  • Schedule C's field, E00900, is in two of the archived lists: the sign-split business line and the AGI default. Both apply.
  • The interest records above the flat-2% value come from the archived structural mortgage conversion, which divides deductible interest by the deductible share where balances exceed the cap (utils/mortgage_interest.py lines 274-283). The 1,243 zeros are the regular records Microcosm's $10M mortgage screen removes. In the processed file itself, before that screen, all 21,765 sit above.
  • 23,817 donor records hold non-qualified dividends where the source's ordinary and qualified dividends are equal, because the two were aged by different factors.

The donor's totals. At its own weights, which total 161.2M against 160.8M SOI returns for TY2021:

SOI line Donor weighted total, $B SOI TY2021, $B SOI TY2023, $B Donor / TY2021 Donor / TY2023
Wages 9,078.3 9,022.4 10,204.1 1.006 0.890
Taxable interest 105.7 103.5 313.8 1.021 0.337
Tax-exempt interest 55.3 55.5 66.1 0.996 0.837
Qualified dividends 297.1 295.9 336.1 1.004 0.884
Capital gain distributions 24.3 23.9 9.3 1.018 2.603
Taxable IRA distributions 406.1 408.4 438.1 0.994 0.927
Taxable pensions 861.8 858.0 932.1 1.004 0.925
Social Security benefits 790.7 791.2 973.6 0.999 0.812
Returns (donor weight total), M 161.18 160.82 160.60 1.002 1.004

The leg that never ran

The archived generator ages TY2015 to TY2021 with uprate_puf (datasets/puf/puf.py line 1379). For a later year its branch at lines 1380-1396 loads the 2021 arrays and runs for variable in uprating: if variable in arrays: … (the loop is lines 1384-1393). uprating is create_policyengine_uprating_factors_table(), a table pivoted to one column per year (utils/uprating.py line 112), so the loop variable is a year and the body never runs. Run unchanged against policyengine-us 2.2.1, the table has columns 2020 to 2034 and the body runs zero times (receipts/run_archived_uprating_loop.txt). The body would also fail as written: it reads a Variable column the pivoted table does not have, and the table names the wage employment_income_before_lsr where the arrays use employment_income.

What that leg would have carried, by the archived method (SOI aggregate growth over return-count growth; the archived table stops at TY2023):

Archived series 2015 to 2021, applied 2021 to 2022 2021 to 2023, not applied
E00200 (employment_income) 1.1871 1.0760 1.1325
E00300 (taxable_interest_income) 1.0105 1.2863 3.0352
E00400 (exempt_interest) 0.8397 0.9977 1.1921
E00600 (ordinary_dividends) 1.3914 1.0622 1.3048
E00650 (qualified_dividends) 1.3628 1.0552 1.1373
E01100 (capital_gains_distributions) 1.9333 0.5367 0.3915
E01000 pos_only (capital_gains_gross) 2.6866 0.6178 0.4722
E01400 (ira_distributions) 1.5092 1.0686 1.0744
E01700 (taxable_pension_income) 1.1637 1.0592 1.0879
E02400 (total_social_security) 1.2234 1.0978 1.2324
E02300 (unemployment_compensation) 7.1791 0.1444 0.1483
E00900 pos_only (business_net_profits) 1.2344 1.0479 1.0271
E00900 neg_only (business_net_losses) 1.6422 1.2573 1.4553
E26270 pos_only (partnership_and_s_corp_income) 1.5313 1.0479 1.0119
REMAINING_VARIABLES (adjusted_gross_income) 1.3560 0.9994 1.0346
S006 weights (return count) 1.0686 1.0032 0.9986

A uniform index from 2021 would be wrong in both directions. Capital gains and their distributions fell by more than half per return after 2021; interest tripled. The partnership and S-corp line is shown for scale: the build donor's partnership column carries neither leg. Unemployment compensation is not a column Microcosm takes from the donor.

What Microcosm does with the donor

Read at main 7639ef8b0 and rechecked at 123927679.

  • tools/build_us_puf_support_base.py::_puf_tax_unit_donor_from_h5 (line 3423) reads the arrays as stored and passes them to puf_tax_unit_donor_from_arrays. Neither that tool, us_runtime/puf_support.py nor us_runtime/puf_qrf_chain.py contains the string "uprat".
  • us/spec/sources.yaml declares kind: uprate, from_year: 2015, to_year_from_build_config: true for the puf_tax_detail stage (lines 230-232). That stage does not run through the source runtime: none of the 26 production run_source_stage call sites in tools/ and the packages passes it. The base tool's one call is for capital_gain_distributions (line 2632), and it builds the PUF donor directly. The US handler map in us_runtime/source_runtime.py has no uprate entry either.
  • The spec pins the build donor's hash and size but gives the 1.8.0 release path as its locator and "2024 (2015-based)" as its vintage (us/spec/sources.yaml lines 50-53 and 132-136). The build donor is 316.9 MB with 97 arrays and 211,677 tax units; the 1.8.0 asset is 241.0 MB with 74 arrays and 207,692.
  • The release tool's repair note for non_sch_d_capital_gains says "the current 2024-level uprating overstates a mean-reverting distribution series" (tools/build_us_fiscal_refresh_release.py line 8281). The column is at its TY2021 level.

What it costs on the Route A release

Three things stand between the donor and a calibration target: the missing leg, the step that carries donor values onto the frame's PUF arm, and the solve. Each is measured below. The first two are of similar size, so the release does not isolate the missing leg.

From the donor to the PUF arm

The release's target-frame checkpoint stores every input per record, after the release's value repairs and before calibration, with the design weights. Each arm starts with half of every household's weight (us_runtime/puf_support.py line 731), so an arm's total times two is the population total that arm implies. The PUF arm here is the PUF clone plus the capital-gains tail clone; at design weights the two together hold exactly the ASEC arm's weight.

Positive amounts only, so that sign-mixed columns do not cancel, with person amounts summed to tax units first:

Donor column Donor at its own weights, $B PUF arm ×2 at design weights, $B PUF arm / donor ASEC arm ×2 at design weights, $B Own series at TY2023 / level the donor carries
Wages 9,078.3 9,860.4 1.09 11,562.0 1.133
Taxable interest 105.7 158.7 1.50 436.2 3.035
Tax-exempt interest 55.3 108.9 1.97 0.0 1.192
Qualified dividends 297.1 363.9 1.22 97.0 1.137
Non-qualified dividends (h) 92.3 137.6 1.49 119.6
Short-term gains (d) 55.0 106.9 1.94 28.9 0.936
Long-term gains (d) 1,112.6 2,225.8 2.00 212.2 0.936
Capital gain distributions (e) 24.3 20.3 0.84 0.0 0.392
Taxable pensions 861.8 1,574.1 1.83 335.6 1.088
Taxable IRA distributions 406.1 730.7 1.80 110.8 1.074
Social Security, retirement component (a) 790.7 1,024.8 1.30 1,198.6 1.232
Schedule C profit, with its SSTB share (b) 699.1 1,024.3 1.47 549.8 0.757
Partnership income, positive (c) 766.2 821.5 1.07 0.0 1.550
Rental income, positive 142.5 277.1 1.95 245.2 0.929
Estate income, positive 48.0 150.7 3.14 0.0 1.020
Cash charitable gifts (f) 196.8 269.7 1.37 0.0 1.040
Real estate taxes (f) 227.3 312.3 1.37 0.0 1.040
Mortgage interest (f) 378.4 476.2 1.26 0.0 1.040
Farm operations income, positive (i) 15.1 17.2 1.14 41.9
Partnership self-employment earnings, positive (i) 106.9 122.0 1.14 0.0
Tax units, M 161.2 184.1 1.14 184.1

(a) Before the solve, the release multiplies each Social Security component on both arms by its SSA target over its design-weight total (0.906 to 2.493 on this build; 1.147 for this component). Since this build, #1178 keeps CPS-reported Social Security on PUF-support rows, so this row describes Route A, not current main.
(b) The donor carries Schedule C's own growth and then AGI growth; its own series alone would put it at 0.757 of the carried level.
(c) The donor carries TY2015 levels; 1.550 is the positive-leg partnership and S-corp line from TY2015 to TY2023.
(d) The donor's Schedule D gains carry AGI growth; the own series here is SOI gross capital gains, positive leg.
(e) After the release rescales this column to its target (factor 0.318). Before that rescale the PUF arm times two was $63.9B, 2.63 times the donor.
(f) The archived method for itemized deductions is the flat 2% a year. SOI itemized totals fell with the number of itemizers after 2017 and are not a per-payer series.
(h) Ordinary less qualified dividends, each aged by its own factor; no single factor applies.
(i) The build calibrates these two columns' mass per unit of weight to the donor's (_PUF_TAX_DETAIL_SIGNED_MASS_CALIBRATED_PERSON_OUTPUTS), and they land on the tax-unit ratio.

Observed: the PUF arm stands for 184.1M tax units where the donor's weights total 161.2M returns, and it carries between 0.84 and 3.14 times the donor's positive mass in these columns. Only the two mass-calibrated columns sit on the tax-unit ratio.

Not established: why. The quantile regression forest (QRF) draw, sparsification to donor positive rates, the capital-gains tail transfer and the release's value repairs all act between the donor and this frame, and this issue did not separate them.

From the PUF arm to the targets

Each calibration target is a compiled column per household. The checkpoint holds those columns, and the calibration file holds the final weights, so every target splits exactly into an ASEC part, a PUF-clone part and a capital-gains-tail part. Summed, the parts reproduce the build's recorded estimates for all 2,955 rows that bind a PUF-sourced variable, to a relative 6e-13.

National dollar rows. Ordinary dividends and taxable Social Security each have two national rows of different vintages that differ by more than 10%; both are shown:

National row, aged to 2024 Source Target, $B PUF arm ×2 / target, design weights ASEC arm ×2 / target, design weights Estimate / target, design weights Estimate / target, final weights PUF-arm part, final / design ASEC-arm part, final / design
Wages TY2023 Table 1.4 10,773.4 0.915 1.073 0.994 0.996 1.255 0.786
Taxable interest TY2023 Table 4.3, excluding dependents 340.4 0.466 1.281 0.874 0.844 1.650 0.717
Tax-exempt interest TY2022 Historic Table 2 61.7 1.765 0.000 0.882 1.004 1.138 n/a
Ordinary dividends TY2023 Table 4.3, excluding dependents 544.8 0.921 0.398 0.659 0.839 1.523 0.694
Ordinary dividends TY2022 Historic Table 2 457.4 1.096 0.474 0.785 0.999 1.523 0.694
Qualified dividends TY2022 Historic Table 2 346.6 1.050 0.280 0.665 0.974 1.670 0.694
Taxable pensions TY2023 Table 1.4 1,013.4 1.553 0.331 0.942 0.989 1.105 0.791
Taxable IRA distributions TY2023 Table 1.4 476.4 1.534 0.233 0.883 1.019 1.180 0.985
Taxable Social Security (a) TY2023 Table 1.4 573.0 1.591 0.694 1.143 0.892 0.870 0.576
Taxable Social Security (a) TY2022 Historic Table 2 510.8 1.785 0.779 1.282 1.000 0.870 0.576
Schedule C net profit TY2022 Historic Table 2 472.3 1.490 1.164 1.327 1.035 0.878 0.655
Partnership and S-corp net income TY2022 Historic Table 2 1,157.6 0.710 0.000 0.355 0.975 2.747 n/a
Net capital gains TY2023 Table 1.4 1,270.9 1.756 0.190 0.973 0.980 1.017 0.923
Capital gain distributions (e) TY2023 Table 1.4 10.2 2.000 0.000 1.000 1.000 1.000 n/a
Estate and trust net income TY2023 Table 1.4 52.1 2.895 0.000 1.447 1.001 0.691 n/a
Rent and royalty net income (g) TY2022 Historic Table 2 95.9 2.049 2.521 2.285 2.640 1.174 1.140
Real estate taxes TY2023 Table 2.1, itemizers 118.1 1.109 0.000 0.555 1.007 1.817 n/a
Mortgage interest TY2023 Table 2.1, itemizers 186.3 1.204 0.000 0.602 1.034 1.719 n/a
Charitable gifts TY2023 Table 2.1, itemizers 230.5 0.966 0.000 0.483 1.137 2.354 n/a
AGI TY2023 Table 1.1 16,619.2 1.118 0.832 0.975 1.004 1.219 0.777

(a) and (e) as above.
(g) Both arms imply more than twice this target at design weights, and the solve does not close it. The donor column is within 8% of its own TY2023 level, so the miss has another cause.

Weights by arm:

Arm Households Weight, design (M) Weight, final (M) Final / design Household ratio: 5th, 50th, 95th percentile Share at the 5x cap
ASEC 168,852 69.05 63.37 0.918 0.09, 0.34, 5.00 6.5%
PUF clone 168,852 68.99 74.69 1.083 0.09, 0.39, 5.00 8.8%
Capital-gains tail clone 15,228 0.06 0.04 0.693 0.25, 0.55, 1.97 1.0%

Observed:

  • The solve moves weight from the ASEC arm to the PUF arm: the PUF clone's total rises 8.3% and the ASEC arm's falls 8.2%.
  • Of the 48 national dollar rows that bind a PUF-sourced variable, the five furthest from target after the solve are rent and royalty net income (2.640, see (g)), taxable interest (0.844 for all returns excluding dependents, 0.480 for itemizers) and ordinary dividends against their TY2023 rows (0.839 and 0.831).
  • Ordinary dividends and taxable Social Security each face two national targets that differ by 19% and 12%: a TY2022 row aged by AGI growth and a TY2023 row. For dividends the TY2023 row is also a different population, returns excluding dependents. The build compiles each pair to one identical household column (as it does the two wage rows), so the solve faces two values for one quantity. It meets the TY2022 row of each (0.999 and 1.000) and misses the TY2023 row (0.839 and 0.892).
  • Taxable interest has no such conflict. Its 51 state rows sum to $334.9B and end at $287.4B (0.858), in line with the national 0.844.
  • Partnership and S-corp income exists only on the PUF arm. It starts at 0.355 of target and reaches 0.975 with the PUF-arm part scaled 2.747 times.
  • Wages start within 1% of target in total, with the PUF arm implying 0.915 of the target and the ASEC arm 1.073. The solve raises the PUF-arm wage part 25.5% and cuts the ASEC part 21.4%.
What a rebase alone would do at design weights

Arithmetic on the design-weight parts above, to TY2023 only. No frame was rebuilt and no solve was run.

National row Estimate / target today PUF arm ×2 / target × own-series factor Estimate / target after
Wages 0.994 0.915 × 1.133 1.055
Taxable interest 0.874 0.466 × 3.035 1.348
Tax-exempt interest 0.882 1.765 × 1.192 1.052
Qualified dividends 0.665 1.050 × 1.137 0.737
Partnership and S-corp net income 0.355 0.710 × 1.550 0.550

On the PUF arm each of these five rows is fed by one donor column. For taxable interest the PUF arm is at 0.466 of target because the donor is at a third of its TY2023 level and the arm carries 1.50 times the donor; aging the column alone overshoots. Pensions, IRA distributions and estate income already exceed their targets on the PUF arm at design weights, and other inputs enter their compiled columns; aging their donor columns up would raise totals that are already high.

Deduced: the missing leg and the donor-to-arm step move the PUF arm's levels by similar amounts, mostly in opposite directions, so correcting the first alone would move some rows toward their targets and others away.

The retired eCPS pipeline

The defect began in the PUF generator the eCPS was built on. In the archived repository's history (policyengine-us-data, every ref of the local clone, which is shallow), the eCPS code took its PUF-imputed amounts from the PUF_2024 class in every version read, and that class's generator never ran the 2021-to-2024 leg.

History, read from the code
  • The loop first appears on 2024-08-19 (d91726f4, "Add uprating tools") in datasets/puf/policyengine_puf.py, in the same commit as the PUF_2024 class; the file became puf.py on 2024-08-23. The table function had returned the pivoted table since 2024-08-18 (737a7924). All 15 versions of utils/uprating.py in the history pivot the table and return it, and all 143 versions of the generator that loop over it open with the same for variable in uprating: line. That is read from the code; the loop was executed only at 42ed5d45.
  • ExtendedCPS_2024 has taken PUF_2024 as its donor since 2024-08-23 (9e759c58). The first complete extended-to-enhanced construction in the history is 2024-08-24 (c28821bc). The archive commit (42ed5d45, 2026-07-02) still binds both extended classes to PUF_2024 (datasets/cps/extended_cps.py lines 1791 and 1800) and trains its imputation on it (calibration/puf_impute.py line 967). At 42ed5d45, none of extended_cps.py, calibration/puf_impute.py and enhanced_cps.py contains the string "uprat".
  • The archived PUF_2021 and PUF_2024 classes point at the published 1.8.0 files (puf.py lines 1645 and 1660), which are equal array for array. On the 1.8.0 puf_2024.h5, all 52 arrays with an archived source expression match the 2015-to-2021 value on every nonzero source record, capital gain distributions among them.

#487 attributes the gap between the build donor's $24.31B of capital gain distributions and the $13.69B it reports for the eCPS to "the current ETL's 2024-level uprating". The two measured donors carry the same ×1.9333 on E01100 on their regular, source-matched records. The donor used for the historical eCPS total was not established, so this comparison cannot identify or exclude the cause of that gap.

Not established: which commit produced the 1.8.0 files, which donor file each eCPS release trained on, and what the stale levels did to any eCPS output. No eCPS artifact was opened and none was rebuilt.

Proposal

Rebase each column from TY2015 by its own series, in Microcosm's donor path.

Where. In the path that builds the donor today, _puf_tax_unit_donor_from_h5 and puf_tax_unit_donor_from_arrays, which already read both the processed file and the TY2015 source and align them by RECID. The spec's uprate declaration would describe that step; adding a handler for it would change nothing, because the stage does not run through the source runtime. The step has to come before the income-rank predictor is computed from the donor's amounts, and the $10M mortgage screen should keep reading the stored values so that its cohort does not move.

For each single-expression dollar column:

  1. Verify the factor the donor carries against the TY2015 source, on every regular record, zeros included. This is the check in this issue, run as a gate.
  2. Multiply by growth(series, 2015, Y) / carried, where Y is the build year and growth is per return.

The result is the TY2015 amount aged once, by its own series, to the build year. A column's "verified price year" is then whatever the gate measured: TY2021 on its own line, TY2021 on the AGI default, TY2015, or TY2021 with a second factor.

Three kinds of column need their own rule:

  • Dividends on regular records. Recover TY2015 ordinary and qualified dividends from the donor's two columns (whole dollars on all 207,692 regular records, at most 2 units in the last place off before rounding), then age qualified by its own line and non-qualified by the ordinary-less-qualified line. Non-qualified is then zero exactly where the source's ordinary equals its qualified, which removes the 23,817 spurious amounts.
  • Dividends on synthetic records. Recovery does not apply: of the 3,376 synthetic records that hold dividends, none recovers to whole TY2015 dollars, 1,343 recover with qualified above ordinary, and 1,190 already store negative non-qualified amounts. Proposed: scale qualified by its line's growth over 1.3628 and non-qualified by its line's growth over the ordinary-dividend factor 1.3914, a convention, which keeps signs (ruling 3).
  • Interest paid, the 11 derived columns and the synthetic records' other columns. The gate cannot verify them. Scale each record by the ratio of its rebased to its carried base amount, which keeps the record's ratio to its base; a record whose carried base is zero is left unchanged. The synthetic records take the same column factors as the regular ones, on a deduced carried factor.

The alternatives:

  • Regenerate the donor. The generator is in an archived repository and its loop needs three repairs. A new file would be frozen at one year again, and would still need the gate to prove what it carries.
  • A 2021-to-Y leg only. Simpler, and it leaves four known errors in place: the two TY2015 columns, Schedule C's second factor and the 23,817 spurious non-qualified amounts.
  • Rebuild the columns from the TY2015 source in Microcosm. For the regular records' single-expression columns this is the same result as the rebase. It has no answer for the synthetic records or the simulated columns, which cannot be rebuilt without rerunning the archived generator's draws.

Which series. The choices that need a ruling:

Columns Proposed series, TY2015 to the latest SOI year, then projected to the build year Open choice
Wages; qualified dividends; Schedule C; partnership and S-corp; net capital gain The line's own SOI actuals, then the matching CBO income-by-source series. us_runtime/target_aging.py already holds this map for targets
Taxable and tax-exempt interest; non-qualified dividends; taxable pensions; IRA distributions; Social Security; capital gain distributions The line's own SOI actuals, then CBO AGI growth, as targets without a CBO series are aged today A closer projection for interest, which tripled from 2021 to 2023
Schedule C, partnership and S-corp, capital gains Separate positive and negative legs, as the archived method does
Schedule D gains with their collectibles and section 1250 parts, which carry AGI growth today SOI net capital gain Keeps the TY2015 shape of a series whose distribution moves (#487)
Rental and estate income, on the AGI default today Their own SOI lines, which the archived table has Or keep AGI
The other 17 AGI-default columns AGI per return
Cash gifts, real estate taxes, interest paid The flat 2% a year Or CPI, or a per-payer series; SOI itemized totals are not one
Per-return divisor SOI return counts to the latest SOI year What divides the projected span: projected returns, or population

Rebasing changes one predictor. The income-rank predictor ranks each donor unit on the sum of six components; components aged by different series reorder that sum.

Invariants

For every build year and every nonnegative weight vector:

  1. The carried factor is verified. For each single-expression column, every regular record, zeros included, equals its TY2015 source value times its leg's carried factor, to a relative 1e-9, or 1e-6 where the stored array is float32. One record off fails the stage, and the stage has no path that skips the check.
  2. The stage runs once. Its output is stamped with the build year and refuses a second rebase.
  3. Identity. A column whose series growth equals its carried factor is unchanged.
  4. Factors chain across the splice. Per-return growth across the last SOI year divides the spliced aggregate's growth by the spliced return count's growth, each counted once at its pivot. Given the same series, the aggregate splice equals the factor target_aging gives a target for the same span.
  5. The source is aged once. Each leg of a single-expression column equals the TY2015 value times growth(series, 2015, Y).
  6. Zeros and signs are preserved on single-expression columns, so the donor positive rates that sparsification reads do not move.
  7. Rank is preserved within a multiplicative single-expression column, within each sign for sign-split columns. Reconstructed non-qualified dividends can reorder.
  8. Each leg's weighted total moves by exactly that leg's factor.
  9. Nothing else moves: weights, identifiers, demographics, flags, origination years.
  10. Dividends. On regular records the recovered TY2015 amounts are whole dollars to within the rounding a division can add (8 units in the last place of the value or of the stored amount it was divided out of; the donor's largest deviation is 2) and below 2^52, qualified never exceeds ordinary, qualified follows its line, and non-qualified follows the ordinary-less-qualified line, is zero exactly where the two are equal, and is never negative. On synthetic records each stored column is scaled by its own factor and keeps its sign.
  11. Derived records keep their ratio to their base, and a record with a zero carried base is unchanged.
  12. Determinism.
  13. The stated carried factors agree with the archived generator, for all 61 fields its lists name and the seven it does not.
Tests
  • Reference tests in proposal/test_rebase_properties.py, against a standalone statement of the semantics in proposal/rebase_reference.py: 24 test functions (19 Hypothesis property tests, 4 fixed checks and one check parametrized over seven values), all passing.
  • Two are differential. Invariant 13 runs the archived uprate_puf on drawn frames of all 68 fields and checks it against a manifest whose factors are recomputed from the SOI table without the archived code. Invariant 4 checks the reference's aggregate splice against target_aging._projection_factor for the same series; the return-count division is tested separately.
  • One strict expected failure states invariant 5 for the uprate declared today, applied to the processed donor. Minimal case: a $1 TY2015 wage is stored as $1.187; aging it by the 2015-to-Y factor again gives $1.187 × g where $1 × g is right.
  • Fourteen faults seeded in the reference are each caught by a test (receipts/mutation_check.txt).
  • These test a reference and sit outside testpaths. In the repo, the gate becomes a build-time check with the donor and source hashes in its receipt, and the property tests move to the package suite against the stage.
  • A regression receipt per build: each column's weighted total over the SOI actual of its line for the latest SOI year, and each column's PUF-arm total over the donor's, the two tables in this issue made permanent.
How it interacts with #1191

#1191 proposes binding the SOI and CBO wage rows to irs_employment_income and storing the PUF-half wage as the donor wage plus the pre-tax amount the engine subtracts.

  • Order in one build: rebase the donor before the QRF, draw, gross up after the draw, bind the rows.
  • The carried donor wage in #1191's proposal becomes the rebased donor wage. Its round-trip invariant is unchanged. Its grid regression check moves to whole TY2015 dollars times growth(wages, 2015, Y).
  • Size. The rebase moves the donor wage by 13.3% to TY2023 before the leg to the build year. The engine's subtraction is $172.2B, 2.78% of PUF-channel wages, in the audit comment on #1191. That comment's fixed-weight illustrations were computed on TY2021-level PUF wages.
  • The rebind without the rebase asks the solve to meet a 2024 tax-return wage target from a PUF half whose donor is at TY2021 levels.
  • The PUF-to-ASEC wage ratio. The audit comment on #1191 measured a median 0.85 for the same person. The missing leg to TY2023 alone is 1 / 1.133 = 0.883. The rest was not decomposed.
  • The release's two value repairs should be revisited with it: the Social Security component rescale and the capital-gain-distribution rescale. The second acts on $31.93B at design weights against a $10.16B target, a factor of 0.318. Aged by 0.3915 the same column would be $12.5B, and the repair would be about 0.81.

The ASEC half has its own vintage question (#981). On this release it pools source years 2022, 2023 and 2024, and its stored wage equals the survey's WSAL_VAL on every ASEC row of each. Rebasing the PUF half makes the two halves consistent only when that is settled too.

Read, observed and deduced

Read in code: the archived 2024 branch and its loop; uprate_puf and its four field lists; the archived source expressions for each donor array; Microcosm's donor loader, its half-weight clone, its handler map, its mass-calibrated column list and the declared uprate.

Observed on the files measured:

  • All 74 arrays of the published 1.8.0 puf_2021.h5 and puf_2024.h5 are equal.
  • On the build donor, 34 single-expression columns and two sums match the archived TY2021 value on their nonzero source records and no other year's (the interest sum on 76.1%). Two more columns equal their raw TY2015 fields, which no hypothesis changes.
  • The donor's weighted totals for eight SOI lines are within 2.1% of the SOI TY2021 actuals and between 0.337 and 2.603 of the TY2023 actuals.
  • The synthetic records' dividend counts and masses above.
  • The archived loop body runs zero times under policyengine-us 2.2.1.
  • The donor-to-arm ratios, the release decomposition and the weight movements in the tables above.

Deduced:

  • That the 3,985 synthetic donor records carry the same factors as the regular ones. They have no source record to compare with. The archived generator ages the file before it disaggregates the aggregate records (puf.py lines 1379 and 1398), and the spec's note says 3,900 of them come from a Forbes-based top tail (us/spec/sources.yaml lines 376-378), whose amounts need not be a source value times a factor.
  • That the 11 derived and simulated columns sit at the level of the amounts they were drawn from.
  • That the published 1.8.0 files were produced by a generator whose loop also did not run. The equal arrays show the result; which commit produced them was not established.
  • What a rebase would do to any target, as above.

Not measured: what a rebuild or a recalibration does under the proposal.

Files measured
File sha256
Build donor puf_2024.h5, Modal volume microcosm-us-stage-inputs 7669f5b5…d6df
TY2015 source puf_2015.csv, same volume 0a7fd643…99df
Published release puf_2021.h5, PolicyEngine/irs-soi-puf 1.8.0 1db51c81…2e45
Published release puf_2024.h5, PolicyEngine/irs-soi-puf 1.8.0 8182579d…f37b
Base frame base_populace_us_2024_puf_support.h5, Route A at 4b57d15a2 0fb05b67…cedd2
Release populace_us_2024.h5, populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z 417d23ae…e1559
Target-frame checkpoint target_frame_checkpoint.h5, same release 8758635b…d28c
Calibration weights populace_us_2024_calibration.npz, same release d1403f75…2d55

The four PUF files are restricted. They were read locally and only aggregates are reported. Full hashes, scripts and receipts: docs/evidence/us-puf-donor-uprating/ (the evidence PR, #1223).

Not done

  • No build, no recalibration, no change to the donor, the spec or any target.
  • The 3,985 synthetic records, the interest-paid columns beyond their 76.1% and the 11 derived columns were not verified record by record.
  • The donor-to-arm ratios are measured and not explained.
  • The leg from TY2023 to the build year is not sized; the archived SOI table stops at TY2023.

Refs #1191, #487, #981.

Dominant language
Python
Stars
0
Forks
5
Avg merge
1d 6h
Merged PRs (30d)
146

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PolicyEngine/microcosm

All issues in PolicyEngine/microcosm

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.