UK: keep constituency and local-authority results available through a switch to Microcosm, then serve them from its local-area dataset

Start by reading _resolve_region, _build_uk_weight_replacement_region, and the related tests, then review issues #727, #717, and PolicyEngine/policyengine.py#553 to establish which routing path applies. Check how runtime_dataset_uri, detect_uk_local_authority_metadata, smoke_app.py, and v2_static_runtime_files_smoke.py handle the routed dataset. Done means UK runs and national-report maps use the appropriate local-area data, routed datasets are validated, and no weight matrices remain for Microcosm local-area regions.

Summary

#727 (closing #725) lets the executor accept a UK dataset without area codes, but only when the policyengine.py bundle declares where constituency and local-authority results come from. Both region types must be routed to another dataset through policyengine.py's regional_dataset_defaults. A missing column on its own is never read as "national-only", so a local-area dataset that lost its area codes still fails.

We don't necessarily intend to publish the Microcosm national dataset ahead of its local-area dataset; #727 keeps that option open in case it becomes necessary. Either way, constituency and local-authority results stay available throughout:

• National and local-area datasets ship together (with policyengine.py routing): constituency and local-authority runs, and the breakdowns in national reports, move straight to the Microcosm local-area dataset.
• The national dataset ships first: those runs and breakdowns keep using the enhanced FRS local-area data until the Microcosm local-area dataset is ready. That means the enhanced FRS file with its area codes, plus its constituency and local-authority weight matrices, which serve them today. The UK default does not switch to a dataset without area codes before this routing is in place.

Omitting the breakdowns is not an acceptable interim state, and #727 does not omit them: on a dataset without area codes, a constituency or local-authority breakdown raises a clear error. A routed national default would therefore fail national reports until the work below lands, so the default does not switch before then.

The executor pins policyengine[models] exactly, so this work lands with, or right after, the bump to the policyengine.py release that changes the UK default. That release is tracked in PolicyEngine/policyengine.py#553; the area-code aliases it needs are in PolicyEngine/microcosm#1114. The check-policyengine-updates workflow opens the bump.

Work

1. Interim routing to the enhanced FRS, needed only if the national dataset ships first.
   • The routing declaration: a UK regional_dataset_defaults in policyengine.py that names the enhanced FRS (as a dataset overlay) for both constituency and local_authority. This is what #727's uk_area_regions_routed reads.
   • Constituency and local-authority runs: keep the weight-matrix regions from _build_uk_weight_replacement_region. Have them load the dataset routed for their region type, resolve_region_dataset_path("uk", region_type), instead of the default; in the interim that is the enhanced FRS. _require_uk_weight_matrix_matches_dataset then confirms each matrix fits it. Keeping the matrices keeps today's numbers.
   • Breakdowns in national reports: replace #727's clear error with a run on the routed dataset, and build the constituency and local-authority outputs from that run. Record which dataset each part of the report came from: the national figures from Microcosm, the local breakdowns from the enhanced FRS.
2. Constituency and local-authority runs on the Microcosm local-area dataset. _resolve_region already asks policyengine.py first (get_region). It only builds a weight-matrix region when policyengine.py returns nothing.
   • If policyengine.py lists the regions by default: the release puts constituency and local-authority regions in its default UK registry, with dataset_path pointing at the local-area dataset (decision 3 in PolicyEngine/policyengine.py#553). The bump alone then moves these runs to row filters on the local file. Remove _build_uk_weight_replacement_region and _require_uk_weight_matrix_matches_dataset once nothing routes to the enhanced FRS, and update their tests.
   • If policyengine.py keeps them opt-in: build the full UK registry here with build_uk_region_registry(include_constituencies=True, include_local_authorities=True).
   • Checks:
     • The local file's revision pin should survive runtime_dataset_uri. It keeps an explicit revision for repositories outside the GCS-mirrored policyengine-*-data repos, which covers populace-uk-private.
     • detect_uk_local_authority_metadata should identify the local file's codes as LAD23.
3. Maps on national reports from the Microcosm local-area dataset. A national run on a file without area codes has nothing to group by. Build these outputs from a second run on the local-area dataset; omitting them is ruled out. Stage 12's geographic aggregate (UK_GEOGRAPHIC_DATASET_VARIABLES) follows the same route.
4. Check the routed dataset at worker start-up.
   • Today: #727 checks a routed dataset when a run loads it, because workers install only the default dataset. Validate the routed dataset's area codes when the worker starts, once workers materialize it.
   • Smoke apps: smoke_app.py and v2_static_runtime_files_smoke.py read the boundary version off the installed default. Point them at the routed dataset when the default carries no area codes.
5. Loading cost. The local-area file's size depends on the Microcosm local release. If it is large, the per-area extracts discussed in #717 for the US local-area file apply here too.
6. Watch the method change.
   • The risk: a policyengine.py release could add constituency and local-authority regions to the default UK registry while the default dataset is still the enhanced FRS, or before the local-area dataset is wired in. The bump would then switch these runs from the calibrated enhanced-FRS weight matrices to row filters on the national file, and the published numbers would change.
   • Mitigation: hold that bump, or keep the weight-matrix route for that release.

Done when

• A UK default without area codes is never live while constituency and local-authority runs, or the breakdowns in national reports, are unavailable. They come from the enhanced FRS in the interim, or from the Microcosm local-area dataset.
• Every dataset that serves constituency or local-authority results is checked for its area codes when the worker starts and when a run loads it.
• UK constituency and local-authority runs on the Microcosm local-area dataset use policyengine.py's regions, with no weight matrices involved.
• National UK reports carry constituency and local-authority maps from the local-area dataset.

Related

• #725 and #727: accepting UK datasets without area codes only when routed, including the release-sequence discussion on #727.
• #717: the US local-area dataset.
• PolicyEngine/policyengine.py#553: certifying a Microcosm national dataset, with separate national and local-area datasets.
• PolicyEngine/policyengine.py#552: regional dataset defaults.
• PolicyEngine/microcosm#1114: _oa aliases on Microcosm local-area exports.

🤖 Generated with Claude Code