UK: track the defects uk-data is fixing that microcosm still has

Start with the update procedure and audit record, then run the listed gh api query for uk-data items above #519. Compare each new item with microcosm main and the open PRs, using the named entry points such as local_targets.py, frs_spine.py, and uc_capital_coherence.py. Done means updating the appropriate checklist, merge state, audit references, and posting a summary comment.

This issue tracks defects that PolicyEngine/policyengine-uk-data PRs and issues fix or report and that microcosm still has. Each review records the last uk-data item it covered. The next review starts at the following number, updates this body, and posts a comment that lists what changed.

Audit record

• Updated: 5 October 2026, after #1100 merged (main 07bbb65). Nine items are ticked and two are added. No new uk-data items were reviewed, so the next review still starts at uk-data#520.
• Reviewed: 2 October 2026.
  • Main pass 12:09–12:40 BST (11:09–11:40 UTC), covering uk-data items up to and including uk-data#516.
  • Addendum 14:10–14:25 BST (13:10–13:25 UTC) for uk-data#517–#519 and for the microcosm items opened in between.
• Last uk-data item covered: uk-data#519, an issue filed 2026-10-02 12:55 UTC. The last PR covered is uk-data#518. The next review starts at uk-data#520.
• Scope: the 32 items filed from uk-data#486 (29 September) to uk-data#519: 17 PRs and 15 issues. The older related issue uk-data#467 is included. Out of scope: uk-data#488 and #489 (LAD lookups, closed unmerged).
• uk-data state:
  • main is at b45c373 (25 September), and none of the items had merged. The last merged code is uk-data#480 (4 September).
  • Of the 17 PRs, only uk-data#490 is out of draft; the other 16 are drafts.
• microcosm refs:
  • main at 3598c38.
  • #1084 (pensions, draft), assessed at 004b5c9 and now at c78dff4.
  • #1089 (#1063 follow-ups, draft, stacked on #1084), head ebff841.
  • #1081 at af6bc17 and #1087 at ec0d0ee.
  • The last microcosm item at review was #1094.
• Code locations: paths are relative to packages/microcosm-build/src/microcosm/build/, with line numbers at main 3598c38.
• Measurements:
  • The calibrated national build from main c5a1cba (30 September): stored 2024-25 columns, calibrated weights, policyengine-uk 2.100.0.
  • Raw FRS 2024-25 and WAS round 8 at design weights where stated.
  • Weighted aggregates only. Cells under 10 records are suppressed.
• Engine: microcosm locks policyengine-uk 2.100.0; PyPI is at 2.104.4. #1086 blocks every release from 2.102.4 on.
• Weight context: the SPI channel holds 47.6% of microcosm's calibrated household weight (13.8m of 29.0m households), against 27.7% for uk-data's SPI half. An SPI-row defect therefore weighs about 1.7 times as much here.

How to update this issue

1. List uk-data items numbered above the last one covered: gh api "repos/PolicyEngine/policyengine-uk-data/issues?state=all&sort=created&direction=desc&per_page=30".
2. Check each new item against microcosm main and the open microcosm PRs, and add it to the right section below.
3. Re-check the merge state of the uk-data and policyengine-uk PRs named here.
   • Tick an item once microcosm main has the fix, and name the PR.
   • Move an engine item up once its policyengine-uk change is in the lock.
4. Update the audit record (date, time, last item covered, refs) and post a comment summarising the changes.

Fix in microcosm now: no policyengine-uk change needed

• FRS dividends are overwritten by an SPI draw. uk-data#498 (part of uk-data#495).
  • Fixed by #1100: FRS respondents keep their reported dividends.
  • Where: hmrc_spi_income_spine redraws dividend_income for every FRS-channel person aged 16 or over from the SPI stage-1 forest, which predicts from age, gender and region only.
    • Code: uk/spec/sources.yaml:1605-1609, uk_runtime/spi_income.py:641-672.
    • Pinned at uk_runtime/spi_spine.py:204 and in test_uk_spi_income.py. #683 adopted the redraw for parity with the incumbent.
  • Measured:
    • FRS-channel dividends are £30.2bn, against £4.0bn reported by the same people.
    • UC-reporting benefit units hold £4.07bn against about £0 reported, and 74.9k of them hold over £20k.
    • policyengine-uk 2.100.0 counts dividends as UC unearned income.
  • Already correct: the other half of uk-data#498, keying the FRS accounts on person_id (uk_runtime/frs_spine.py:538-571).
  • Fix:
    • Remove the base-row redraw, then rebuild and recalibrate.
    • Receipts: the 24 HMRC dividend bands, UC by channel, and WAS corporate wealth.
    • Land it before any engine bump that brings pe-uk#1950.
• 16–19-year-old dependants in SPI rows get taxpayer earnings. uk-data#504.
  • Fixed by #1100: SPI draws go only to FRS claimants and partners, so dependants keep their twin's values.
  • Already fixed: under-16s, by #879 (uk_runtime/spi_income.py:74).
  • Measured: the guard is age-only.
    • 93% of SPI-channel 16–17-year-olds have earnings (£11.0bn), against 4% on FRS rows.
    • Dependants aged 16–19 in non-advanced education carry £12.7bn.
    • £0.95bn of it sits in UC-reporting benefit units.
  • Fix: exclude FRS dependent children (16–19, in non-advanced education, inside a parent's benefit unit) from the stage-1 recipients. Bundle it with the dividend fix so one rebuild covers both.
• council_tax is the bill after council tax reduction. uk-data#496 (fixed there by uk-data#499).
  • Fixed by #1100: council_tax is the bill before the reduction, and Scottish water and sewerage are netted at the gross cells (65% for recipients, ruling of 2 October). The charge paid follows the reduction scheme's combined maximum: the larger of the status discount and 35%. Still open: María's signatures on the two register texts in #1100's description, and the #953/#736 corrections below.
  • Where:
    • uk_runtime/frs_council_tax.py:70-101 keeps CTANNUAL, which the FRS 2024-25 DV summary defines as "Annual CT amount after discounts/reduction".
    • A test pins CTANNUAL = 0 at £0.
    • The imputation pool includes recipients' net bills.
    • council_tax_rebate (CTREBAMT × 365.25/7, uk_runtime/frs_spine.py:805-807) already holds the add-back, but nothing adds it.
  • Measured, FRS channel:
    • 20.6% of English CTR reporters have a £0 bill, and 25.6% a bill below their own CTR (Wales: 14.8% and 18.4%).
    • The add-back totals £0.78bn in England, £0.05bn in Wales and £0.06bn in Scotland.
    • obr.council_tax binds council_tax_less_benefit, so CTR comes off twice: about £0.5bn for English working-age reporters. The UK row ends at −10.4%.
  • Scotland:
    • microcosm nets sewerage, which uk-data main does not, but at the wrong rate: uk-data#499's status-discount check gives 0.81–0.89 here, against 0.75 with the gross DVs.
    • The water-charge helper treats CWATAMTD/CWATAMT1 as the discount factor. That ratio averages 0.86 for undiscounted households, so Scottish water_and_sewerage_charges run about 13–16% low. This is a residual of uk-data#467; #747 fixed the zeroing itself.
  • Fix:
    • Add back CTREBAMT for CTREB = 1, including CTANNUAL = 0.
    • Where the reduction is unknown, use max(CTANNUAL, cell mean).
    • Impute from non-recipients only.
    • Net Scottish water and sewerage from the gross DVs. Ruling needed: 65% for recipients.
    • Give the water charge its own rule.
    • #953 and #736's CT-C item describe the FRS bill as gross of CTR. Both need correcting.
• OBR table 4.9 benefit rows are measured over the UK. Follow-up to uk-data#490.
  • Fixed by #1100: each row binds Great Britain or England and Wales, and obr.pip binds pip + dla over England and Wales (ruling of 2 October). The pension-age HB spending row and the Chronicle stamp now have their own item below.
  • Where:
    • These rows in uk/uk_population_targets.json carry no country condition: obr.attendance_allowance, carers_allowance, pip, winter_fuel_allowance, pension_credit, esa, jobseekers_allowance, universal_credit, statutory_maternity_pay and housing_benefit.
    • dwp.esa_* lacks a GB filter.
    • DWP's lines cover GB. They cover England and Wales only for CA from September 2018, DLA/PIP/AA from April 2020 and Winter Fuel from April 2024.
  • Measured: share of each fitted UK column outside the publisher's scope: AA 10.8%, CA 12.3%, PIP 8.9%, ESA 7.5%, PC 4.7%, UC 2.3%.
  • Fix:
    • Add region household_conditions, as dwp.uc.* and dwp.hb.* already use.
    • Fold PC, AA and Winter Fuel into #1084. Its final run shows obr.attendance_allowance moving from 0.0% to +2.6% against the new AA caseload rows.
    • Ask PolicyEngine/chronicle to stamp table 4.9's DWP lines GB or England and Wales.
    • Add DWP's GB pension-age HB spending; it is already in feed 825406f.
  • Ruling needed: obr.pip binds PIP alone against OBR's combined DLA and PIP line.
    • Its note defers to #622, which closed without a ruling.
    • Calibration lifts PIP from £18.9bn to £36.1bn.
• Scottish ADP and CDP benefit codes are not mapped. uk-data#500.
  • Fixed by #1100.
  • Where: BENEFIT_CODES (uk_runtime/frs_spine.py:131-152) has no code 117/118 (Adult Disability Payment) or 121/122 (Child Disability Payment).
  • Measured:
    • Raw FRS: 215k ADP daily-living recipients (£0.98bn), 147k ADP mobility, 51k CDP care and 27k CDP mobility.
    • 73% of Scottish daily-living recipients are dropped. PIP daily-living reporters per 1,000 residents: Scotland 24.6, England 66.0.
  • Fix:
    • Map ADP into the pip_* and CDP into the dla_* reported columns. Do it with the country-scope item, or about £1.4bn of Scottish spending lands in a UK-measured PIP column fitted to an England and Wales line.
    • Codes 69 and 70 are benefit recoveries; leave them unmapped.
• SPI rows carry benefit claims nobody observed. Remainder of uk-data#514.
  • Fixed by #1100: the new spi_benefit_coherence stage and its release-blocking gate. It also zeroes income-related ESA, and keeps contributory ESA.
  • Already ahead of uk-data:
    • UC, through uc_reporter_redraw;
    • HB and CTR, through the housing shell (#1012);
    • contributory ESA on earners: 3.7% of SPI reporters here, against 41% in uk-data.
  • Still open:
    • 1.51m SPI benefit units claim UC only because their FRS donor reported it. The OR refresh in uk_runtime/uc_capital_coherence.py:154-158 never removes a donor's anchor.
    • receives_benefits_in_own_right (uk_runtime/frs_education.py:193-198) disagrees with the person's own reports for 3.04m SPI persons, 61k of them aged 16–19.
    • SPI rows report IS, WTC, CTC and income-based JSA at 3.75, 3.0, 2.6 and 9.1 times the FRS rate.
    • Contributory JSA is 17 times the FRS rate; it is already signed out under dwp.jsa_claimants (#869).
    • SPI rows hold 64%, 72% and 67% of reported IIDB, AFCS and bereavement support (£0.63bn, £0.74bn and £1.06bn).
  • Fix: one stage after uc_capital_coherence. It should:
    • restore the donor's IIDB, AFCS and bereavement support;
    • zero contributory JSA, SDA, IS, tax credits, income-based JSA and SSMG;
    • decide income-related ESA against the bound dwp.esa_* rows;
    • rebuild receives_benefits_in_own_right;
    • redraw UC take-up: reporters True, everyone else at the residual rate within uc_age_eligible.
  • Don't port uk-data#514's HB/CTR zeroing or its PC/CB rules; #1084 and #1089 supersede them.
• Local targets and predictors use the age-18 child. uk-data#486.
  • Fixed by #1100 for the constituency UC bands, the ETB predictors and the WAS donor. Still open: the LCFS variable names, and María's signature on the reworded uc_unit_vs_household_grain adjudication and census fence.
  • Already correct: the national targets, which use UC's child or qualifying young person and the ONS dependent child.
  • Constituency UC child bands use the age-18 num_children (uk_runtime/local_targets.py:282-289), bound under adjudication uc_unit_vs_household_grain. 2.77% of UC-reporting units fall in a different band. Fix in #882's lane.
  • ETB VAT and services pair the engine's is_adult/is_child with ETB's own counts (uk_runtime/etb_vat.py:34-37, uk_runtime/etb_services.py:42-57). 2.3% of households mismatch.
  • WAS: the donor's NumCh18R8 overlaps NumAdultR8 in 8% of households (uk_runtime/was_wealth.py:36-57). Fix with #145's predictor revisit.
  • LCFS: the counts already agree; only the variable names need changing.
• Buy-to-let is missing from other_residential_property_value. Part of uk-data#501 and uk-data#495.
  • Fixed by #1100 (ruling of 2 October), with rental_income added as a WAS predictor.
  • Where: uk_runtime/was_wealth.py:173 maps DVHseValR8 only. Buy-to-let (DVBltValR8, £573bn in WAS round 8) is left out, and stays out in #1089.
  • Measured: with buy-to-let included, 91% of rent-receiving WAS households hold the matching property, against 25% now. This reaches the HB, PC and legacy capital tests.
  • Fix in #1089 / #1063: carve buy-to-let into the column (ruling needed on the concept), and add rental income as a predictor for the property segment.
• SPI benefit units' capital ignores their capital income. uk-data#495.
  • Fixed by #1100: every SPI unit's capital is redrawn conditioned on its investment income, with a receipt of the units still incoherent.
  • Where: uc_capital_coherence redraws TOTCAPB4 only for post-fill UC reporters, by children and couple status (uk_runtime/uc_capital_coherence.py:141,217-239).
  • Measured: 2.28m SPI benefit units have interest and dividends that imply over £16k of capital at a 4% yield, but TOTCAPB4 of £16k or less. 105k of them are UC reporters.
  • Fix: condition the redraw on investment income, extend it beyond reporters, and add a per-release receipt of income held without a matching capital stock.
• Property income: in review as #1081. uk-data#503, uk-data#509.
  • Port verified: it gives the same property income as uk-data#509's rule on all 34,966 FRS 2024-25 person records.
  • CI fails only test_no_incumbent_data_package_references_in_live_tree: the changelog fragment and the test docstring cite the incumbent package by name instead of uk-data#503/#509.
  • Follow-up: uk_runtime/cgt_asset_type.py:752-754 reads property_income > 0 as evidence of residential property, so the 42 loss-making landlords lose that signal after #1081.
• BRMA weights: in review as #1087 (CI green). uk-data#515, uk-data#516.
  • Main's draw reproduces the copied lists. Scottish shares: Lothian 7.4% and Greater Glasgow 2.0%, against 19.3% and 15.9% in the 2022 census.
  • #1081 and #1087 share generated files; regenerate them on the second merge.
  • Related: #1090 and #1092 (Scottish private-rent targets at BRMA grain).
• HBAI validation: fold into #731. uk-data#487.
  • Correction: the #968 hbai_basis compares engine year 2024 with HBAI FYE 2024. Period 2024 is FRS FY2024/25 (docs/uk-uc-paid-target-contract.md), which is HBAI FYE 2025.
  • Against FYE 2025: the v21c scorecard sits inside HBAI's 95% confidence intervals for relative poverty (14.63% before housing costs, 19.31% after). Its median is 6.7% above HBAI's.
  • Fix (validation only, never bound):
    • re-base to FYE 2025;
    • report each statistic as inside or outside its interval;
    • source the tables through a PolicyEngine/chronicle package that records vintages, since DWP revises FYE 2022–25 in March 2027.
• Reproducibility hardening. uk-data#507.
  • The core defect is absent here: microcosm's dense solve has no dropout, the targets are sha-pinned, and the draws are identity-keyed.
  • Remaining:
    • record the torch thread count;
    • give the HardConcrete gates their own torch.Generator;
    • add one calibrated-weights twin to the next national build.
  • #1089 keys the SPI band donors' positional draw (uk_runtime/spi_band_donors.py:354-380).
• Access-fund period code. From Vahid's review of #1100.
  • Where: one FRS 2024-25 access-fund award carries the calendar-month period code (ACCSSPD 5). Once annualised, it is eleven times the next largest FRS value, and it is almost certainly an annual or termly award. The detail is in the retired entry in uk/input_mass_reviewed_exclusions.json, as it stood before a0c678c21.
  • Why it needs an item:
    • #1089 recorded the defect as a signed input-mass exclusion, after calibration took the household to the 10× weight bound.
    • #1100 retired the exclusion: on its measurement arm the column sits at +62% of the reference, inside the gate's fence, so the gate fails the entry as stale.
    • The record itself is unchanged, so any calibration that weights the household up again brings the overshoot back.
  • Fix: a period rule for the access fund in the FRS spine (uk_runtime/frs_spine.py), so the column stops depending on where calibration lands.
• GB pension-age Housing Benefit spending row. Dropped from #1100.
  • dwp.hb.amount_pension_age cannot compile. calendar_year_window needs both fiscal years from one series identity, but the DWP forecast-table facts in feed 825406f carry one measure_id per year (expenditure_2024, expenditure_2025).
  • Needs: a PolicyEngine/chronicle series-identity change, or a ruling to bind FY2025-26 alone. Raise it together with the request that Chronicle stamp table 4.9's DWP lines GB or England and Wales.

Depends on policyengine-uk model updates

microcosm locks policyengine-uk 2.100.0. Since 2.102.4, policyengine-uk requires a policyengine-core that the certified US default refuses, so no release from 2.102.4 on can be locked until #1086 is resolved. Each item names the policyengine-uk change it needs and that change's state on 2 October 2026.

• Legacy-benefit reporters never move to UC. uk-data#492.
  • Needs:
    • a policyengine-uk input for claiming UC at legacy closure (uk-data#492's option b; not yet filed);
    • income-related ESA made exclusive with UC;
    • pe-uk#1910, working-age HB abolition (open).
  • Measured:
    • 552.6k working-age benefit units report HB, tax credits, IS, income-based JSA or income-related ESA and have would_claim_uc False: 313.6k on FRS rows and 239.1k on SPI rows.
    • 2.100.0 already closes tax credits, IS and income-based JSA, so those reporters lose the benefit today.
    • Income-related ESA is paid whatever the flag says.
  • Why the data alone can't fix it: anchoring in the data (option a) fails here, because HB and CTC require ~would_claim_uc in the 2025 calibration year.
  • Next step: bring DWP's Move to UC claim rates by cohort in through PolicyEngine/chronicle. They are the new evidence that #796 asks for before the frozen 0.55 rate changes.
• Rent liability for sharers, sub-tenants, boarders and lodgers. uk-data#511, uk-data#512, uk-data#508, uk-data#506.
  • Needs: pe-uk#2006, pe-uk#1995 and pe-uk#2002, all draft.
  • Shared households: rent is the whole dwelling's HHRENT, placed on the household head (uk_runtime/frs_spine.py:829). About 685k calibrated benefit units in shared households (353k on FRS rows, 332k on SPI rows) have no rent of their own.
  • Sub-letting: SUBRENT sits in property_income. microcosm's own household subrent column is redrawn for SPI copies and read by nothing. Ruling needed: does SPI INCPROP already include rent-a-room receipts?
  • Boarders and lodgers: CVPAY, the rent they pay (£0.81bn), is not carried.
  • Plan: port all three as one change at the bump.
• Pension Credit reported capital. uk-data#513.
  • Needs: pe-uk#2018 (open).
  • Fix: set pension_credit_reported_capital to frs_benunit_capital (TOTCAPB4, which equals TOTCAPB3 for 81% of PC reporters).
  • Smaller here than in uk-data, because microcosm already strips pension wealth from corporate_wealth. Deemed income from the engine's capital reaches the reported PC for 11.6% of FRS-row PC reporters, against 1.7% with FRS capital.
• Severe disability flag. uk-data#494.
  • Needs: pe-uk#1946 (draft, stacked on pe-uk#1896).
  • Where: uk_runtime/frs_disability.py:282-284 uses the old definition. 0.55m of the 4.93m people it flags fail the tax-credit condition.
  • Plan: change it in lockstep with the engine, because on 2.100.0 the flag still drives the severe disability premium.
• Secured debt and the share/trust split. uk-data#501.
  • Needs:
    • pe-uk#1969 (open) for the secured-debt inputs;
    • engine variables for directly_held_shares, unit_and_investment_trusts, cash_isa and stocks_and_shares_isa, which have no policyengine-uk PR yet.
  • Reconcile with #1063's mortgage rule: mortgage_debt is HMortGR8, which includes £234bn secured on other property. Choose between all secured debt and main residence only (TotMortR8, £1,014bn), so the debt is not counted twice.
• Property in UC capital. uk-data#495.
  • Needs:
    • pe-uk#1950 (open);
    • a policyengine-uk change, not yet filed, so that dataset-reported capital adds property rather than replacing it.
  • Why: TOTCAPB4 holds no property, so once pe-uk#1950 lands no landlord benefit unit meets UC's property test.
  • Order: land the dividend fix above first.
• Person-type variables. uk-data#486.
  • Needs: pe-uk#1896 (open).
  • At that bump:
    • export is_hbai_dependent_child and the claimant-or-partner flag from the FRS roles;
    • update the UC take-up population in uk_runtime/frs_take_up.py:51-62, which mirrors the old is_uc_eligible. Its lockstep test pins the deprecated is_WA_adult, so the drift would pass silently.
• Mixed-age couple Pension Credit saving. uk-data#519.
  • Needs: pe-uk#1940 (open).
  • Why: microcosm stores no birth year and holds ages fixed across model years. The model's default test (older member born 1954 or earlier, computed as model year minus age) therefore drifts one cohort a year.
  • Fix: set has_mixed_age_couple_pension_credit_saving at build time from survey-year facts.
• Marriage Allowance. uk-data#502.
  • Needs: pe-uk#1963 (draft).
  • Current state: microcosm reads neither MAIND nor PAS; its take-up flag is a Bernoulli(0.5) draw.
  • At the bump: re-measure recipients against HMRC's 2.35m.
  • For #840: the proposed MAIND ↔ is_married bridge is wrong-sided, because MAIND flags recipients only.
• Published LHA rates. pe-uk#2022 (draft). The locked engine computes LHA rates from its own copied list of rents, so #1087's weights reach published rates only after this bump.
• Pension-age Housing Benefit new claims. pe-uk#1901, merged and released in 2.102.5; locking it needs #1086.
• Director dividends and owner-managers.
  • Needs: pe-uk#1965 (draft) for the owner-manager inputs.
  • Data gap: neither repository reads adult DIVIDNET or job DIVIDEND (£12–20bn a year net).

Already handled in microcosm

These need no new work and are listed so later reviews skip them.

• Take-up after the SPI chain, uk-data#491:
  • UC: through the OR refresh on main.
  • Pension Credit: in #1084 (98.6k SPI units are not re-anchored on main).
  • Child Benefit: in #1089 (248.8k on main).
• SPI rows' CTR, uk-data#497: one household draw on the reference person since #1012.
• No UC flag on all-pension-age units, the take-up half of uk-data#490: excluded from the UC draw since #937.
• State Pension level, uk-data#493: main's build has £127.0bn on 11.49m recipients, and #1084's final run gives £142.5bn against a £142.3bn target.
• Pension Credit take-up over entitled units, uk-data#510: #1084 solves it by component. Two tweaks remain there: a GB-only solve, and a rule for how units a reform newly entitles claim.
• Pension wealth and ISA split, uk-data#501: done by #750, #681 and #180.
• Scottish water zeroing, uk-data#467: fixed by #747. The rate residual sits under the council tax item above.
• Area draws within the household's region, uk-data#517: region-constrained by default in uk_runtime/rowwise_geography.py, and no caller turns it off.
• Salary sacrifice for people without pay, uk-data#518: main's build has 1.94m such people carrying £7.08bn, and #1089 restricts the prediction to paid people. Fewer than 10 records sacrifice more than their pay.
• Accounts dividends keyed on person_id, the keying half of uk-data#498: already correct.

Not applicable

• uk-data#505: microcosm never simulates the SPI tape, and region UNKNOWN exists only on the donor table. One optional check: composite and address-abroad donors hold 18% of SPI dividends.