suite: a section counts PASS: and FAIL: anywhere in a line of a child, and one failed check of a child is counted as two

未关闭
#2,382 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
42/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
c, shell
领域
testing

调研方向

Start with tests/run_all_tests.sh and inspect the 30 PASS: counting sites and 33 FAIL: counting sites, then read the child scripts’ output and exit-status handling. Reproduce both planted cases in the issue, including exit 3 before a child’s final line. Done when check counts distinguish child checks from runner errors, incomplete children are still reported, and release and ASan suite totals are explained.

由索引模型根据 Issue 内容生成。

描述

area:gates kind:bug

Found during the review of #2379 (the second pass of its blind review), and true before it: every line below was run with the tests/run_all_tests.sh of 68dbb67e (phase2-numbers), byte for byte, on a make http build. main has the same lines; it was not run there.

What happens

Thirty sections of tests/run_all_tests.sh run a child script, keep its output and count its checks with grep -c "PASS:"; thirty-three count failures with grep -c "FAIL:". Neither pattern has an anchor, so every line that holds the text counts as a check, whoever printed it and whatever else the line says.

1. A line that is not a check is counted as one. One line added at the end of tests/test_http_rss_growth.sh, in a copy:

echo "    note (planted): an earlier run said PASS: RSS9"
[45c/47] HTTP per-request leak gate (RSS growth)
  PASS: all 5 HTTP RSS-growth checks
  RESULTS: 194/194 passed, 0 failed, 1 skipped

Run by itself with the same line planted, the child's own last line is HTTP_RSS: 4 passed, 0 failed, and the suite's run without the line counts 193.

This is how the first commit of #2379 read "all 5" for 4 checks, green in every lane: its new check ran a second program and printed that program's own PASS: line beside its own. The second commit stopped printing that line. Nothing stops the next one: the blind review put the line back and nothing went red.

2. A child that reports one failed check is counted as two failures, and is told that it did not complete. One failing check added in the same place, in a copy (fail "RSS9 planted failing check"):

[45c/47] HTTP per-request leak gate (RSS growth)
  FAIL: 2 HTTP RSS-growth check(s) failed
  FAIL: RSS9 planted failing check
  FAIL: test_http_rss_growth.sh exited 1 without completing — section verdict is not trustworthy (#988)
  RESULTS: 193/195 passed, 2 failed, 1 skipped

The second FAIL: line is the runner's own, for a child that exits with a status other than 0 (#988), and the section counts it with the child's. For a child that died it is the right line. This child prints its tally (HTTP_RSS: 4 passed, 1 failed when it is run by itself with that plant) and ends with [ "$FAIL" -eq 0 ], as the suite's child scripts do: it completed, and it exited 1 because a check failed. Whoever reads a red lane sees two failures for one failed check, and is pointed at a crash that did not happen.

Why it matters

The counts are read. A hand check before a merge quotes them ("all 4 HTTP RSS-growth checks", "5188/5188 passed"), and a section's count going up is taken as a check added. A number that can go up without a check, and a failure line that says "did not complete" of a script that did, are both small, and both are in the instrument that every other result is read from.

Done when

  • With the first plant, the section's count is the child's own (4), or the section is red and names the line it could not account for. Shown for 45c and for two more of the thirty sections, chosen by the pull request.
  • With the second plant, the suite's RESULTS: line counts one failure, and no line says that the child did not complete.
  • A child that exits with a status other than 0 before it printed its last line is still reported as not completed, and counted, as #988 asks. Shown by a plant (exit 3 in the middle of a child).
  • The thirty sites and the thirty-three decide in one place, not in sixty-three copies of a pattern.
  • A release run and an ASan run of the whole suite give the same RESULTS: totals before and after, apart from sections whose count was wrong before; the pull request lists each of those with its reason.

— Claude DevBox

主要语言
C
星标
3
派生
7
平均合并
3 小时 28 分钟
30 天内合并 PR
270

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

InauguralSystems/EigenScript 的其他 Issue

查看 InauguralSystems/EigenScript 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。