selftest: count-unspec-canary is seed-dependent and goes red ~0.24% of selftests on a correct tree

Aperta
#72 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
lua, shell
Ambito
testing-qa

Direzione di ricerca

The flake comes from corpus/lua-tests/math.lua line 845, which calls math.randomseed() with no argument, so the count-unspec-canary plant in tools/selftest.sh (lines 79-82) grades lua-count with --repeat 20 against a floor set in tools/grade.sh line 417. Read the grade floor logic and the self-test row in tests/gut.sh first. Done means the plant's verdict is independent of the time-derived seed, with 1,000 consecutive clean runs or a stated false-red bound, and the row still goes red when the floor is gutted.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Found while timing #71 (hq#58 Wave 2). The selftest plant count-unspec-canary can go red on a correct tree. Main has not changed since c650143.

Observed: run 41349374525 attempt 3, job 124982803346, step RUN selftest. The same head was green on attempts 1 and 2, and the failing step runs before any of #71's changes take effect:

PLANT count-unspec-canary expected=math.lua=PASS nextvar.lua=PASS rc=0 got=math.lua=FAIL nextvar.lua=PASS rc=1 WRONG

Mechanism:

  • corpus/lua-tests/math.lua:845 calls math.randomseed() with no argument, so the seed comes from the time and an address.
  • The statistics loops that follow then run a seed-dependent number of asserts.
  • The plant (tools/selftest.sh:79-82) grades lua-count with --repeat 20.
  • tools/grade.sh:417 sets floor = lo * 0.60, where lo is the minimum of 20 reference runs. Any of the 20 candidate runs that falls below that floor turns the row FAIL.

Measured: 9,000 runs of .ref/bin/lua-count math.lua.

  • Counts ran from a minimum of 25,699 to a maximum of 237,979, with a median of about 78.6k.
  • rc was 0 every time, and the digit-free output had the same md5 every time.
  • Under the plant's rule, that distribution gives an estimated ~0.24% false red per selftest, about 1 in 420.
  • In the collector's 84 earlier suite logs it was never red, which fits that rate.
  • Load does not change it. The distribution measured while a 4-lane fuzz ran was the same as without it (KS D=0.0147, 5% critical value 0.0304).

This is a gate that can go red without a bug. It was not a liar in the false-green direction, but a flaky plant trains people to rerun red.

Done when:

  • The plant's verdict no longer depends on the time-derived seed. For example: fixed-seed or fixed-count inputs for the plant, or a floor taken from a reference sample with a stated false-red bound of 1e-6 or less per selftest. Do not edit the vendored test, and do not lower the 0.60 fraction as the fix.
  • Evidence in the PR: at least 1,000 consecutive runs of the plant's grade command (tools/grade.sh --candidate .ref/bin/lua-count --ref .ref/bin/lua --tier S --only math.lua,nextvar.lua --repeat 20) with 0 FAIL, or the computed bound.
  • The count-unspec-canary row in tests/gut.sh still bites: with the floor gutted, the self-test goes red.

Generated by Claude Code

Lingua principale
Lua
Stelle
0
Fork
0
Merge medio
43m
PR unite (30g)
10

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di InauguralSystems/EigenLua

Tutte le issue di InauguralSystems/EigenLua

Issue simili

Altre issue su Lua

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.