selftest: count-unspec-canary is seed-dependent and goes red ~0.24% of selftests on a correct tree

オープン
#72 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
22/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
lua, shell
領域
testing-qa

調査の方向性

The flake comes from corpus/lua-tests/math.lua line 845, which calls math.randomseed() with no argument, so the count-unspec-canary plant in tools/selftest.sh (lines 79-82) grades lua-count with --repeat 20 against a floor set in tools/grade.sh line 417. Read the grade floor logic and the self-test row in tests/gut.sh first. Done means the plant's verdict is independent of the time-derived seed, with 1,000 consecutive clean runs or a stated false-red bound, and the row still goes red when the floor is gutted.

索引モデルが issue の本文から書いたものです。

説明

Found while timing #71 (hq#58 Wave 2). The selftest plant count-unspec-canary can go red on a correct tree. Main has not changed since c650143.

Observed: run 41349374525 attempt 3, job 124982803346, step RUN selftest. The same head was green on attempts 1 and 2, and the failing step runs before any of #71's changes take effect:

PLANT count-unspec-canary expected=math.lua=PASS nextvar.lua=PASS rc=0 got=math.lua=FAIL nextvar.lua=PASS rc=1 WRONG

Mechanism:

  • corpus/lua-tests/math.lua:845 calls math.randomseed() with no argument, so the seed comes from the time and an address.
  • The statistics loops that follow then run a seed-dependent number of asserts.
  • The plant (tools/selftest.sh:79-82) grades lua-count with --repeat 20.
  • tools/grade.sh:417 sets floor = lo * 0.60, where lo is the minimum of 20 reference runs. Any of the 20 candidate runs that falls below that floor turns the row FAIL.

Measured: 9,000 runs of .ref/bin/lua-count math.lua.

  • Counts ran from a minimum of 25,699 to a maximum of 237,979, with a median of about 78.6k.
  • rc was 0 every time, and the digit-free output had the same md5 every time.
  • Under the plant's rule, that distribution gives an estimated ~0.24% false red per selftest, about 1 in 420.
  • In the collector's 84 earlier suite logs it was never red, which fits that rate.
  • Load does not change it. The distribution measured while a 4-lane fuzz ran was the same as without it (KS D=0.0147, 5% critical value 0.0304).

This is a gate that can go red without a bug. It was not a liar in the false-green direction, but a flaky plant trains people to rerun red.

Done when:

  • The plant's verdict no longer depends on the time-derived seed. For example: fixed-seed or fixed-count inputs for the plant, or a floor taken from a reference sample with a stated false-red bound of 1e-6 or less per selftest. Do not edit the vendored test, and do not lower the 0.60 fraction as the fix.
  • Evidence in the PR: at least 1,000 consecutive runs of the plant's grade command (tools/grade.sh --candidate .ref/bin/lua-count --ref .ref/bin/lua --tier S --only math.lua,nextvar.lua --repeat 20) with 0 FAIL, or the computed bound.
  • The count-unspec-canary row in tests/gut.sh still bites: with the floor gutted, the self-test goes red.

Generated by Claude Code

主要言語
Lua
スター
0
フォーク
0
平均マージ
43分
マージ済み PR(30日)
10

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

InauguralSystems/EigenLua のほかの issue

InauguralSystems/EigenLua の issue をすべて見る

似ている issue

Lua の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。