selftest: count-unspec-canary is seed-dependent and goes red ~0.24% of selftests on a correct tree

未关闭
#72 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
22/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
lua, shell
领域
testing-qa

调研方向

The flake comes from corpus/lua-tests/math.lua line 845, which calls math.randomseed() with no argument, so the count-unspec-canary plant in tools/selftest.sh (lines 79-82) grades lua-count with --repeat 20 against a floor set in tools/grade.sh line 417. Read the grade floor logic and the self-test row in tests/gut.sh first. Done means the plant's verdict is independent of the time-derived seed, with 1,000 consecutive clean runs or a stated false-red bound, and the row still goes red when the floor is gutted.

由索引模型根据 Issue 内容生成。

描述

Found while timing #71 (hq#58 Wave 2). The selftest plant count-unspec-canary can go red on a correct tree. Main has not changed since c650143.

Observed: run 41349374525 attempt 3, job 124982803346, step RUN selftest. The same head was green on attempts 1 and 2, and the failing step runs before any of #71's changes take effect:

PLANT count-unspec-canary expected=math.lua=PASS nextvar.lua=PASS rc=0 got=math.lua=FAIL nextvar.lua=PASS rc=1 WRONG

Mechanism:

  • corpus/lua-tests/math.lua:845 calls math.randomseed() with no argument, so the seed comes from the time and an address.
  • The statistics loops that follow then run a seed-dependent number of asserts.
  • The plant (tools/selftest.sh:79-82) grades lua-count with --repeat 20.
  • tools/grade.sh:417 sets floor = lo * 0.60, where lo is the minimum of 20 reference runs. Any of the 20 candidate runs that falls below that floor turns the row FAIL.

Measured: 9,000 runs of .ref/bin/lua-count math.lua.

  • Counts ran from a minimum of 25,699 to a maximum of 237,979, with a median of about 78.6k.
  • rc was 0 every time, and the digit-free output had the same md5 every time.
  • Under the plant's rule, that distribution gives an estimated ~0.24% false red per selftest, about 1 in 420.
  • In the collector's 84 earlier suite logs it was never red, which fits that rate.
  • Load does not change it. The distribution measured while a 4-lane fuzz ran was the same as without it (KS D=0.0147, 5% critical value 0.0304).

This is a gate that can go red without a bug. It was not a liar in the false-green direction, but a flaky plant trains people to rerun red.

Done when:

  • The plant's verdict no longer depends on the time-derived seed. For example: fixed-seed or fixed-count inputs for the plant, or a floor taken from a reference sample with a stated false-red bound of 1e-6 or less per selftest. Do not edit the vendored test, and do not lower the 0.60 fraction as the fix.
  • Evidence in the PR: at least 1,000 consecutive runs of the plant's grade command (tools/grade.sh --candidate .ref/bin/lua-count --ref .ref/bin/lua --tier S --only math.lua,nextvar.lua --repeat 20) with 0 FAIL, or the computed bound.
  • The count-unspec-canary row in tests/gut.sh still bites: with the floor gutted, the self-test goes red.

Generated by Claude Code

主要语言
Lua
星标
0
派生
0
平均合并
43 分钟
30 天内合并 PR
10

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

InauguralSystems/EigenLua 的其他 Issue

查看 InauguralSystems/EigenLua 的全部 Issue

相似的 Issue

更多 Lua Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。