selftest: count-unspec-canary is seed-dependent and goes red ~0.24% of selftests on a correct tree

Ouverte
#72 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
22/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
lua, shell
Domaine
testing-qa

Piste de recherche

The flake comes from corpus/lua-tests/math.lua line 845, which calls math.randomseed() with no argument, so the count-unspec-canary plant in tools/selftest.sh (lines 79-82) grades lua-count with --repeat 20 against a floor set in tools/grade.sh line 417. Read the grade floor logic and the self-test row in tests/gut.sh first. Done means the plant's verdict is independent of the time-derived seed, with 1,000 consecutive clean runs or a stated false-red bound, and the row still goes red when the floor is gutted.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Found while timing #71 (hq#58 Wave 2). The selftest plant count-unspec-canary can go red on a correct tree. Main has not changed since c650143.

Observed: run 41349374525 attempt 3, job 124982803346, step RUN selftest. The same head was green on attempts 1 and 2, and the failing step runs before any of #71's changes take effect:

PLANT count-unspec-canary expected=math.lua=PASS nextvar.lua=PASS rc=0 got=math.lua=FAIL nextvar.lua=PASS rc=1 WRONG

Mechanism:

  • corpus/lua-tests/math.lua:845 calls math.randomseed() with no argument, so the seed comes from the time and an address.
  • The statistics loops that follow then run a seed-dependent number of asserts.
  • The plant (tools/selftest.sh:79-82) grades lua-count with --repeat 20.
  • tools/grade.sh:417 sets floor = lo * 0.60, where lo is the minimum of 20 reference runs. Any of the 20 candidate runs that falls below that floor turns the row FAIL.

Measured: 9,000 runs of .ref/bin/lua-count math.lua.

  • Counts ran from a minimum of 25,699 to a maximum of 237,979, with a median of about 78.6k.
  • rc was 0 every time, and the digit-free output had the same md5 every time.
  • Under the plant's rule, that distribution gives an estimated ~0.24% false red per selftest, about 1 in 420.
  • In the collector's 84 earlier suite logs it was never red, which fits that rate.
  • Load does not change it. The distribution measured while a 4-lane fuzz ran was the same as without it (KS D=0.0147, 5% critical value 0.0304).

This is a gate that can go red without a bug. It was not a liar in the false-green direction, but a flaky plant trains people to rerun red.

Done when:

  • The plant's verdict no longer depends on the time-derived seed. For example: fixed-seed or fixed-count inputs for the plant, or a floor taken from a reference sample with a stated false-red bound of 1e-6 or less per selftest. Do not edit the vendored test, and do not lower the 0.60 fraction as the fix.
  • Evidence in the PR: at least 1,000 consecutive runs of the plant's grade command (tools/grade.sh --candidate .ref/bin/lua-count --ref .ref/bin/lua --tier S --only math.lua,nextvar.lua --repeat 20) with 0 FAIL, or the computed bound.
  • The count-unspec-canary row in tests/gut.sh still bites: with the floor gutted, the self-test goes red.

Generated by Claude Code

Langage dominant
Lua
Étoiles
0
Forks
0
Merge moyen
43 min
PR mergées (30 j)
10

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de InauguralSystems/EigenLua

Toutes les issues de InauguralSystems/EigenLua

Issues similaires

Plus d'issues Lua

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.