Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Benchmark gate: optional valid-task fraction floor, so one flaky lm-eval task does not fail a multi-task run — deferred until a real run reports a flaky-task false failure

Abierto
  • ¿Algo no está bien?
#914 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
testing

Línea de trabajo

Start with BenchmarkConfig and run_benchmark, where any invalid lm-eval task score currently fails the whole gate, and with the deferral comment there. Use evaluation.llm_judge.min_valid_fraction as the pattern for an opt-in floor. Done means the floor is off by default, reports excluded tasks, the row at docs/roadmap/risks-and-decisions.md:258 is removed, and the CHANGELOG names the ID. The issue says its trigger is not met, so confirm with a maintainer before starting.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area: eval-gates deferred severity: low source: roadmap wave: 4

Deferred work · F-W20260729-CORE-03-floor · severity Low · cost ~2-3 h · recorded in docs/roadmap/risks-and-decisions.md:258

Summary

Phase 16 S2 (decision C-1) made the benchmark gate fail when any lm-eval task returns a non-finite or out-of-range score, so under auto_revert: true one flaky task in a ten-task run deletes the model. A min_valid_fraction-style floor mirroring evaluation.llm_judge.min_valid_fraction was deferred so the strict form's real cost is known before a second threshold is added to a gate that used to pass NaN.

Condition or budget

Revisit if a real run reports a flaky-task false failure. Not met as of 2026-10-09.

State today (re-checked 2026-10-09)

BenchmarkConfig has no valid-fraction field; run_benchmark fails the whole gate on any invalid task score, and its comment records the floor as considered and deferred.

Related

#118 (no "could not evaluate" outcome for infrastructure failures) and #120 (would make missing or no-accuracy tasks fail closed too).

Acceptance criteria

  • An opt-in floor excludes a minority of unusable task scores and reports them, while the default stays strict.
  • The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).

Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.

Lenguaje dominante
Python
Estrellas
9
Forks
1
Merge medio
11 h 8 min
PR fusionados (30 d)
10

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de HodeTech/ForgeLM

Todos los issues de HodeTech/ForgeLM

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.