Batch rerun on a completed run dir replays early rounds and regresses the CSV
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start by tracing the eval-no-tools-batch resume path and how persisted state under batches/ is read and written. Compare the stored round provenance with the regenerated by_model/.csv using results/local/gpt55_default_run_20260705_attempt1/. Done means rerunning a completed directory is a no-op or resumes from the last persisted round without increasing missing predictions.
Written by the indexing model from the issue text.
Description
Problem
Rerunning eval-no-tools-batch against an output dir whose run already completed does not extend it — it replays the stored batch state from round 0 ("round 1: resuming batch batch_…"), stops before the later rounds, and overwrites by_model/<model>.csv with fewer repair rounds applied, silently discarding the completed run's later repairs.
Observed (gpt-5.5 default-effort run, 2026-07-05):
- First invocation: rounds 0–5, final CSV 1,984 rows with 23 missing predictions.
- Second invocation of the identical command on the same dir (intending to continue repairs with
POLICYBENCH_MAX_REPAIR_ROUNDS=8): re-collected rounds 0–2 from the stored OpenAI batch ids, printed "scheduling repair round" after round 2, then wrote the CSV and exited — leaving 51 missing predictions (rounds 3–5's successful repairs discarded).
Whether the raised POLICYBENCH_MAX_REPAIR_ROUNDS (#97) was honored by the resume path is also unclear from the logs — the process stopped scheduling after round 2 despite 18 units still violating.
Expected
Either (a) a completed run dir is a no-op ("already complete: 1,984 rows, 23 missing — pass --force to redo"), or (b) resume continues from the LAST persisted round and only schedules new repair rounds for currently-missing units, never producing a CSV with more missing cells than the one it replaces.
Workaround used
Archived the dir and ran fresh in a new output dir. Evidence preserved at results/local/gpt55_default_run_20260705_attempt1/ (both CSVs' round provenance in batches/).
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 11h 11m
- Merged PRs (30d)
- 22
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PolicyEngine/policybench
-
v2 prompt and data: program-specific disability facts, integer inputs, and the take-up boundaryOpen
Difficulty 5/5 Over a week Newbie friendliness 38/100
PolicyEngine/policybench#165 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
PolicyEngine/policybench#145 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 52/100
PolicyEngine/policybench#140 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
PolicyEngine/policybench#139 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
PolicyEngine/policybench#129 ·
Maintainers usually reply within 1 day
All issues in PolicyEngine/policybench
Similar issues
-
python triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
microsoft/semantic-kernel#14590 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
pinterest/pymemcache#662 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 Half a day Newbie friendliness 62/100
gnocchixyz/gnocchi#1527 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
tenstorrent/tt-model-manager#181 ·
Maintainers usually reply within 1 day