Retry run-time dataset downloads, and move latest only after a deploy's checks

未关闭
  • 哪里不对？
#1,022 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

@anth-volk 已经在做这个了。

开始于 2026年11月6日。

  • #1023 来自 @anth-volk —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
停滞
技术栈
github-actions, google-cloud, python
领域
backend-api-design, ci-cd, devops

调研方向

Start with the failed release run and inspect the runtime dataset download in policyengine.py 6.12.1, then trace the routing finalizer, "Publish routing after all services pass," and simulation-deployment-ready.sh in the deployment workflow. Check linked pull request #1023 for current work. Done means downloads retry only the specified transient failures, routing publication and latest promotion follow the described order, and rerunning failed jobs completes the deploy.

由索引模型根据 Issue 内容生成。

描述

Release run 42040665530 (attempt 1) failed one staging integration test, test_calculate_uk_local_area_breakdown, because the executor timed out reading the UK local-area dataset from Hugging Face's CDN (requests.exceptions.ConnectionError: ... Read timed out.). A dataset other than the image's default is downloaded into a temporary folder by each new container, and policyengine.py 6.12.1 makes a single request per file with no retry.

Because a check failed, the routing finalizer restored staging's routing and recorded "restored". Attempt 2 ("Re-run failed jobs") passed every check, but the job "Publish routing after all services pass" had succeeded in attempt 1 and did not run again, so the finalizer returned attempt 1's recorded "restored", and simulation-deployment-ready.sh, which requires "completed", failed. Re-running only the failed jobs cannot complete such a deploy.

What should change

  1. Retry the executor's run-time dataset load with bounded backoff on connection errors, read timeouts and 5xx responses, never retrying 401, 403, 404 or a SHA-256 mismatch, and never reusing a partial download.
  2. Split the routing publication:
    • Before the checks, publish only the release's routes (its policyengine.py version and its US and UK model versions, mapped to the deployed app) and its bundle manifest, leaving latest.
    • After every check and the Cloud Run promotion, move latest when the release is newer, check that unpinned requests reach the release, and put latest back if that check fails.
    • "Re-run failed jobs" then completes a deploy without republishing the routes, and no "restored" outcome from an earlier attempt remains.
主要语言
Python
星标
1
派生
1
平均合并
1 天 23 小时
30 天内合并 PR
89

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

PolicyEngine/policyengine-sim-api 的其他 Issue

查看 PolicyEngine/policyengine-sim-api 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。