Retry run-time dataset downloads, and move latest only after a deploy's checks

Aperta
  • Qualcosa non va?
#1,022 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@anth-volk ci sta già lavorando.

Dal 6/11/2026.

  • #1023 di @anth-volk — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
github-actions, google-cloud, python
Ambito
backend-api-design, ci-cd, devops

Direzione di ricerca

Start with the failed release run and inspect the runtime dataset download in policyengine.py 6.12.1, then trace the routing finalizer, "Publish routing after all services pass," and simulation-deployment-ready.sh in the deployment workflow. Check linked pull request #1023 for current work. Done means downloads retry only the specified transient failures, routing publication and latest promotion follow the described order, and rerunning failed jobs completes the deploy.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Release run 42040665530 (attempt 1) failed one staging integration test, test_calculate_uk_local_area_breakdown, because the executor timed out reading the UK local-area dataset from Hugging Face's CDN (requests.exceptions.ConnectionError: ... Read timed out.). A dataset other than the image's default is downloaded into a temporary folder by each new container, and policyengine.py 6.12.1 makes a single request per file with no retry.

Because a check failed, the routing finalizer restored staging's routing and recorded "restored". Attempt 2 ("Re-run failed jobs") passed every check, but the job "Publish routing after all services pass" had succeeded in attempt 1 and did not run again, so the finalizer returned attempt 1's recorded "restored", and simulation-deployment-ready.sh, which requires "completed", failed. Re-running only the failed jobs cannot complete such a deploy.

What should change

  1. Retry the executor's run-time dataset load with bounded backoff on connection errors, read timeouts and 5xx responses, never retrying 401, 403, 404 or a SHA-256 mismatch, and never reusing a partial download.
  2. Split the routing publication:
    • Before the checks, publish only the release's routes (its policyengine.py version and its US and UK model versions, mapped to the deployed app) and its bundle manifest, leaving latest.
    • After every check and the Cloud Run promotion, move latest when the release is newer, check that unpinned requests reach the release, and put latest back if that check fails.
    • "Re-run failed jobs" then completes a deploy without republishing the routes, and no "restored" outcome from an earlier attempt remains.
Lingua principale
Python
Stelle
1
Fork
1
Merge medio
1g 23h
PR unite (30g)
89

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di PolicyEngine/policyengine-sim-api

Tutte le issue di PolicyEngine/policyengine-sim-api

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.