Retry run-time dataset downloads, and move latest only after a deploy's checks

Ouverte
  • Quelque chose ne va pas ?
#1,022 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 2 jours

@anth-volk y travaille déjà.

Depuis le 6/11/2026.

  • #1023 par @anth-volk — ouverte

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
35/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
À l'abandon
Stack technique
github-actions, google-cloud, python
Domaine
backend-api-design, ci-cd, devops

Piste de recherche

Start with the failed release run and inspect the runtime dataset download in policyengine.py 6.12.1, then trace the routing finalizer, "Publish routing after all services pass," and simulation-deployment-ready.sh in the deployment workflow. Check linked pull request #1023 for current work. Done means downloads retry only the specified transient failures, routing publication and latest promotion follow the described order, and rerunning failed jobs completes the deploy.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Release run 42040665530 (attempt 1) failed one staging integration test, test_calculate_uk_local_area_breakdown, because the executor timed out reading the UK local-area dataset from Hugging Face's CDN (requests.exceptions.ConnectionError: ... Read timed out.). A dataset other than the image's default is downloaded into a temporary folder by each new container, and policyengine.py 6.12.1 makes a single request per file with no retry.

Because a check failed, the routing finalizer restored staging's routing and recorded "restored". Attempt 2 ("Re-run failed jobs") passed every check, but the job "Publish routing after all services pass" had succeeded in attempt 1 and did not run again, so the finalizer returned attempt 1's recorded "restored", and simulation-deployment-ready.sh, which requires "completed", failed. Re-running only the failed jobs cannot complete such a deploy.

What should change

  1. Retry the executor's run-time dataset load with bounded backoff on connection errors, read timeouts and 5xx responses, never retrying 401, 403, 404 or a SHA-256 mismatch, and never reusing a partial download.
  2. Split the routing publication:
    • Before the checks, publish only the release's routes (its policyengine.py version and its US and UK model versions, mapped to the deployed app) and its bundle manifest, leaving latest.
    • After every check and the Cloud Run promotion, move latest when the release is newer, check that unpinned requests reach the release, and put latest back if that check fails.
    • "Re-run failed jobs" then completes a deploy without republishing the routes, and no "restored" outcome from an earlier attempt remains.
Langage dominant
Python
Étoiles
1
Forks
1
Merge moyen
1 j 23 h
PR mergées (30 j)
89

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de PolicyEngine/policyengine-sim-api

Toutes les issues de PolicyEngine/policyengine-sim-api

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.