Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Migrate US staging run files to the version 2 contract

Open
  • Is something wrong?
#1,135 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start by reading the version 2 writer introduced by #895, the US fiscal-refresh and exact-count paths from #933, and the telemetry separation in #1099. Trace their writer, fixtures, lifecycle, diagnostics, and delivery tests, then verify US version 2 reads in Calibration Diagnostics before changing the producer. Done means all listed acceptance criteria pass, including removal of the version 1 writer and tests for each run and delivery mode.

Written by the indexing model from the issue text.

Description

Problem

The US fiscal-refresh build still writes schema version 1 staging run files, while UK builds use the stricter, country-neutral version 2 format introduced by #895. Both implementations write the same conceptual run bundle, but Microcosm maintains separate writer classes, fixtures, lifecycle behavior, and dashboard parsing paths.

PR #1099 separates hosted event delivery through LocalTelemetryEmitter from staging run-file persistence. It intentionally leaves the existing US file format unchanged. The next step should migrate US builds to version 2 and remove the version 1 writer.

Version 2 should be the single current staging run-file contract because it provides:

  • named and validated document schemas;
  • country, operation, pipeline, candidate, release, and run-kind identity;
  • explicit lifecycle and delivery state;
  • ordered events and consistent bundle validation;
  • reviewed aggregate-only artifacts with digests and size limits;
  • safeguards against uploading row-level data, credentials, archives, or dataset files;
  • optional authenticated remote read-back.

Required changes

  • Use the version 2 writer for every US fiscal-refresh build, including exact-count builds covered by #933.
  • Replace the two writer classes with one country-neutral StagingRunBundleWriter implementation based on the current version 2 code.
  • Stop producing schema version 1 staging files.
  • Remove the version 1 writer, producer fixtures, and producer-only tests after the US path migrates.
  • Keep LocalTelemetryEmitter functionally independent. Disabling staging run files with --no-staging must not disable hosted event delivery.
  • Populate the version 2 country, operation, pipeline, candidate, release, run-kind, sampling, and delivery fields for US builds.
  • Convert US lifecycle statuses and events to the version 2 contract.
  • Audit every US diagnostic currently attached to the staging run. Version 1 accepts arbitrary JSON files; version 2 permits only reviewed aggregate data and rejects record arrays, sensitive keys, dataset files, and oversized files. Summarize or omit artifacts that do not satisfy the version 2 content policy.
  • Verify that Calibration Diagnostics reads new US version 2 runs before deploying the producer change. Retaining read-only support for historical version 1 runs is a separate compatibility decision and does not require retaining a version 1 writer.

Acceptance criteria

  • US fiscal-refresh and exact-count builds instantiate the single shared staging run-bundle writer.
  • No production Microcosm path emits schema version 1 staging files.
  • Every US staging document and attached artifact passes version 2 schema and content validation.
  • US run identity and delivery fields are complete and consistent across the manifest, progress document, event stream, calibration progress, and run index.
  • A staging file write, validation, or upload failure cannot suppress hosted events from LocalTelemetryEmitter.
  • --no-staging suppresses only staging run files and their uploads.
  • Calibration Diagnostics tests cover a US version 2 run before the Microcosm producer change is deployed.
  • Tests cover successful, failed, local-only, remote-delivery-failure, and exact-count US runs.
  • The obsolete version 1 writer and its producer fixtures are removed.

Context

  • #895 introduced the version 2 staging run-file contract for UK builds while explicitly retaining version 1 for US builds.
  • #933 covers enabling staging output for US exact-count builds.
  • #1099 separates the always-on hosted event emitter from staging run-file persistence.
Dominant language
Python
Stars
0
Forks
5
Avg merge
1d 19h
Merged PRs (30d)
115

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PolicyEngine/microcosm

All issues in PolicyEngine/microcosm

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.