selftest: count-unspec-canary is seed-dependent and goes red ~0.24% of selftests on a correct tree

Abierto
#72 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
22/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
lua, shell
Área
testing-qa

Línea de trabajo

The flake comes from corpus/lua-tests/math.lua line 845, which calls math.randomseed() with no argument, so the count-unspec-canary plant in tools/selftest.sh (lines 79-82) grades lua-count with --repeat 20 against a floor set in tools/grade.sh line 417. Read the grade floor logic and the self-test row in tests/gut.sh first. Done means the plant's verdict is independent of the time-derived seed, with 1,000 consecutive clean runs or a stated false-red bound, and the row still goes red when the floor is gutted.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Found while timing #71 (hq#58 Wave 2). The selftest plant count-unspec-canary can go red on a correct tree. Main has not changed since c650143.

Observed: run 41349374525 attempt 3, job 124982803346, step RUN selftest. The same head was green on attempts 1 and 2, and the failing step runs before any of #71's changes take effect:

PLANT count-unspec-canary expected=math.lua=PASS nextvar.lua=PASS rc=0 got=math.lua=FAIL nextvar.lua=PASS rc=1 WRONG

Mechanism:

  • corpus/lua-tests/math.lua:845 calls math.randomseed() with no argument, so the seed comes from the time and an address.
  • The statistics loops that follow then run a seed-dependent number of asserts.
  • The plant (tools/selftest.sh:79-82) grades lua-count with --repeat 20.
  • tools/grade.sh:417 sets floor = lo * 0.60, where lo is the minimum of 20 reference runs. Any of the 20 candidate runs that falls below that floor turns the row FAIL.

Measured: 9,000 runs of .ref/bin/lua-count math.lua.

  • Counts ran from a minimum of 25,699 to a maximum of 237,979, with a median of about 78.6k.
  • rc was 0 every time, and the digit-free output had the same md5 every time.
  • Under the plant's rule, that distribution gives an estimated ~0.24% false red per selftest, about 1 in 420.
  • In the collector's 84 earlier suite logs it was never red, which fits that rate.
  • Load does not change it. The distribution measured while a 4-lane fuzz ran was the same as without it (KS D=0.0147, 5% critical value 0.0304).

This is a gate that can go red without a bug. It was not a liar in the false-green direction, but a flaky plant trains people to rerun red.

Done when:

  • The plant's verdict no longer depends on the time-derived seed. For example: fixed-seed or fixed-count inputs for the plant, or a floor taken from a reference sample with a stated false-red bound of 1e-6 or less per selftest. Do not edit the vendored test, and do not lower the 0.60 fraction as the fix.
  • Evidence in the PR: at least 1,000 consecutive runs of the plant's grade command (tools/grade.sh --candidate .ref/bin/lua-count --ref .ref/bin/lua --tier S --only math.lua,nextvar.lua --repeat 20) with 0 FAIL, or the computed bound.
  • The count-unspec-canary row in tests/gut.sh still bites: with the floor gutted, the self-test goes red.

Generated by Claude Code

Lenguaje dominante
Lua
Estrellas
0
Forks
0
Merge medio
43 min
PR fusionados (30 d)
10

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de InauguralSystems/EigenLua

Todos los issues de InauguralSystems/EigenLua

Issues similares

Más issues de Lua

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.