Benchmark gate: optional valid-task fraction floor, so one flaky lm-eval task does not fail a multi-task run — deferred until a real run reports a flaky-task false failure
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start with BenchmarkConfig and run_benchmark, where any invalid lm-eval task score currently fails the whole gate, and with the deferral comment there. Use evaluation.llm_judge.min_valid_fraction as the pattern for an opt-in floor. Done means the floor is off by default, reports excluded tasks, the row at docs/roadmap/risks-and-decisions.md:258 is removed, and the CHANGELOG names the ID. The issue says its trigger is not met, so confirm with a maintainer before starting.
由索引模型根据 Issue 内容生成。
描述
Deferred work ·
F-W20260729-CORE-03-floor· severity Low · cost ~2-3 h · recorded indocs/roadmap/risks-and-decisions.md:258
Summary
Phase 16 S2 (decision C-1) made the benchmark gate fail when any lm-eval task returns a non-finite or out-of-range score, so under auto_revert: true one flaky task in a ten-task run deletes the model. A min_valid_fraction-style floor mirroring evaluation.llm_judge.min_valid_fraction was deferred so the strict form's real cost is known before a second threshold is added to a gate that used to pass NaN.
Condition or budget
Revisit if a real run reports a flaky-task false failure. Not met as of 2026-10-09.
State today (re-checked 2026-10-09)
BenchmarkConfig has no valid-fraction field; run_benchmark fails the whole gate on any invalid task score, and its comment records the floor as considered and deferred.
Related
#118 (no "could not evaluate" outcome for infrastructure failures) and #120 (would make missing or no-accuracy tasks fail closed too).
Acceptance criteria
- An opt-in floor excludes a minority of unusable task scores and reports them, while the default stays strict.
- The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).
Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.
- 主要语言
- Python
- 星标
- 9
- 派生
- 1
- 平均合并
- 8 小时 47 分钟
- 30 天内合并 PR
- 47
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
HodeTech/ForgeLM 的其他 Issue
-
area: site bug good first issue severity: low wave: 4
难度 2/5 1-3 小时 新手友好度 75/100
HodeTech/ForgeLM#961 · 1 条评论 ·
维护者通常 1 天内回复
-
area: dev-tooling bug severity: low source: roadmap wave: 4
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
Trend-tracking jq recipe prints evaluation_completed=false as 'n/a', because jq's // operator treats false like null (F-W1-DOCS-TRAIN-16)可能已有人在做 @yuefdev 于 4 天前认领。 未关闭documentation good first issue severity: low source: review-2026-09 wave: 4
难度 1/5 1-3 小时 新手友好度 80/100
维护者通常 1 天内回复
-
documentation severity: medium source: review-2026-09 wave: 3
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
documentation good first issue severity: medium source: review-2026-09 wave: 3
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
相似的 Issue
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 78/100
yunaremaia/proto-drift#22 ·
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 82/100
yunaremaia/memwatch#96 ·
-
bug good first issue
难度 1/5 1 小时以内 新手友好度 91/100
yunaremaia/ci-gate-watch#34 ·
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 85/100
yunaremaia/agent-capability-attestation#64 ·
维护者通常 1 天内回复
-
bug good first issue
难度 1/5 1 小时以内 新手友好度 86/100
yunaremaia/agent-call-graph#43 ·
维护者通常 1 天内回复