Stage 12 coordinator can outlive its timeout while waiting and leaves workers running
Maintainers usually reply within 1 day
@anth-volk is already working on this.
Since Sep 30, 2026.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 20/100
Research direction
The issue does not name the coordinator's source file, so start by finding where the Stage 12 coordinator waits on the baseline and reform workers, using the 3,000 s wait and 4,500 s Modal timeout as search terms. Check the existing pull request #793 before starting, since it already targets this work. Done means both waits share one deadline that leaves time to save the outcome, and a failed or timed-out worker cancels the unfinished sibling and records it as failed.
Written by the indexing model from the issue text.
Description
Problem
The Stage 12 coordinator can be killed before it records a failure, and it leaves workers running after a comparison has failed.
- Separate wait timeouts: the coordinator waits up to 3,000 s for the baseline worker, then up to a fresh 3,000 s for the reform worker. Its own Modal timeout is 4,500 s. When workers are queued, the two waits can exceed that, and Modal kills the coordinator before it saves the failure.
- Siblings keep running: when one worker's wait fails or times out, the other worker keeps running. That can mean a 32 GB container for up to 50 minutes on a comparison that has already failed.
One production report from 6 to 13 October 2026 waited 3,000.0175 s for baseline and then stored TimeoutError. Its reform worker had already been interrupted after about 305 s.
Expected
- Both waits share one deadline that leaves the coordinator time to save the outcome before its Modal timeout.
- When one worker fails or times out, the coordinator cancels every unfinished worker and records the cancelled one as failed.
Observing baseline and reform concurrently is out of scope here; it is part of #712.
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 40
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PolicyEngine/policyengine-sim-api
-
Integration tests fail due to missing UV virtual environment setupPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
PolicyEngine/policyengine-sim-api#314 ·
Maintainers usually reply within 1 day
-
Update policyengine to 6.4.3Possibly taken @policyengine claimed this today. Open
Difficulty 1/5 1-3 hours Newbie friendliness 35/100
PolicyEngine/policyengine-sim-api#812 ·
Maintainers usually reply within 1 day
-
Update policyengine to 6.4.2Possibly taken @policyengine claimed this 1 day ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 40/100
PolicyEngine/policyengine-sim-api#810 ·
Maintainers usually reply within 1 day
-
Register weighted-microdata rules: weights aren't valuesPossibly taken @anth-volk claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 30/100
PolicyEngine/policyengine-sim-api#806 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
PolicyEngine/policyengine-sim-api#805 ·
Maintainers usually reply within 1 day
All issues in PolicyEngine/policyengine-sim-api
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbacalso24/mimry#291 ·
Maintainers usually reply within 1 day
-
good first issue
Difficulty 2/5 Under an hour Newbie friendliness 82/100
eli-alvarado/us-watchlist-alerts#14 ·
Maintainers usually reply within 1 day
-
from-review-extraction python severity:nit
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
LearningCircuit/local-deep-research#7462 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
cathrynlavery/diagram-design#441 ·
Maintainers usually reply within 2 days
-
[Cleanup] Include the public header in work_split.cppPossibly taken @blozano-tt claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
tenstorrent/tt-metal#60850 ·
Maintainers usually reply within 1 day