Benchmark gate: optional valid-task fraction floor, so one flaky lm-eval task does not fail a multi-task run — deferred until a real run reports a flaky-task false failure
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 25/100
Research direction
Start with BenchmarkConfig and run_benchmark, where any invalid lm-eval task score currently fails the whole gate, and with the deferral comment there. Use evaluation.llm_judge.min_valid_fraction as the pattern for an opt-in floor. Done means the floor is off by default, reports excluded tasks, the row at docs/roadmap/risks-and-decisions.md:258 is removed, and the CHANGELOG names the ID. The issue says its trigger is not met, so confirm with a maintainer before starting.
Written by the indexing model from the issue text.
Description
Deferred work ·
F-W20260729-CORE-03-floor· severity Low · cost ~2-3 h · recorded indocs/roadmap/risks-and-decisions.md:258
Summary
Phase 16 S2 (decision C-1) made the benchmark gate fail when any lm-eval task returns a non-finite or out-of-range score, so under auto_revert: true one flaky task in a ten-task run deletes the model. A min_valid_fraction-style floor mirroring evaluation.llm_judge.min_valid_fraction was deferred so the strict form's real cost is known before a second threshold is added to a gate that used to pass NaN.
Condition or budget
Revisit if a real run reports a flaky-task false failure. Not met as of 2026-10-09.
State today (re-checked 2026-10-09)
BenchmarkConfig has no valid-fraction field; run_benchmark fails the whole gate on any invalid task score, and its comment records the floor as considered and deferred.
Related
#118 (no "could not evaluate" outcome for infrastructure failures) and #120 (would make missing or no-accuracy tasks fail closed too).
Acceptance criteria
- An opt-in floor excludes a minority of unusable task scores and reports them, while the default stays strict.
- The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).
Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.
- Dominant language
- Python
- Stars
- 9
- Forks
- 1
- Avg merge
- 8h 10m
- Merged PRs (30d)
- 5
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HodeTech/ForgeLM
-
area: dev-tooling bug severity: low source: roadmap wave: 4
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
area: site bug good first issue severity: low source: review-2026-09 wave: 4
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
-
documentation good first issue severity: medium source: review-2026-09 wave: 3
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
documentation good first issue severity: low source: review-2026-09 wave: 4
Difficulty 1/5 1-3 hours Newbie friendliness 80/100
-
documentation severity: medium source: review-2026-09 wave: 3
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
All issues in HodeTech/ForgeLM
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 1/5 Under an hour Newbie friendliness 62/100
-
Difficulty 1/5 Under an hour Newbie friendliness 70/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
topoteretes/cognee#5773 ·
Maintainers usually reply within 1 day
-
MiKTeX: a failed build reports no errors because texify --quiet switches the log to file:line: styleOpen
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
SublimeText/LaTeXTools#1718 ·