fix(ci): contract cells and the mutants job fail intermittently with exit 4 (the server under test never comes up)

未关闭
  • 哪里不对？
#93 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
38/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
github-actions, rust, shell
领域
ci-cd, devops, testing

调研方向

Start with scripts/contract.sh around pick_port and the exit-4 paths, then read scripts/mutants.sh around mutant output and summary handling. Check .github/workflows/flake-hunt.yml and contract-cell.yml for the affected jobs, and the referenced AGENTS.md rules 2 and 8. Done means server logs are visible for non-killed mutants, startup failures are distinct from survivors, and repeated runs meet the agreed acceptance criteria.

由索引模型根据 Issue 内容生成。

描述

area:process bug prio:p1

Context

scripts/contract.sh exits 4 when the server under test never comes up: "target did not become healthy" or "target process exited early" (contract.sh:221 and :226). Contract cells and the mutants job then fail intermittently. Because gate requires every job, one of these turns the required check red, on main and on pull requests, for a reason that says nothing about the code.

scripts/mutants.sh runs each mutant as contract.sh ... > "$OUT/$name.out" 2>&1 (:53), so the server log that contract.sh prints on exit 4 goes into a file the job log does not show. Its summary then labels any non-kill SURVIVED or did not run (exit=..) (:33), so a server that never started reads like a surviving mutant, which is the finding the job exists to make.

Evidence

Where Job What was seen
push to main, b1c9d1f (run 40258128717) mutants native-memory engine + case-sensitive-search SURVIVED or did not run (exit=4, failed tests=)
#86, first run (42158481836) mutants native-memory main + ignore-revoked ... (exit=4, failed tests=)
#86, first run contract (worker-cf, bindings) Address already in use (127.0.0.1:34322), then target process exited early, exit 4
#88, a throwaway pull request (42164639829) mutants ignore-revoked again, exit=4; the other ten mutants were killed
#88 contract (worker-cf, main) Address already in use (127.0.0.1:43896), exit 4

Of the nine pushes to main from ad07a79 to a206319, four failed (ae7dfa6, c0736a0, 7e4e858, b1c9d1f). For c0736a0, 7e4e858 and b1c9d1f the failing job was mutants ("Every mutant must be KILLED"); I did not inspect ae7dfa6. The last three pushes passed.

It is not a real surviving mutant: rerunning the failed jobs of #86 passed, #87's own run was green, and running the ignore-revoked mutant locally on the same tree gave a healthy server and a kill (49 passed, 5 failed, exit 101).

What is not known

  • Why mutants fails: its log does not show the server's. The worker-cf cells are Address already in use.
  • Two hypotheses, neither tested. (1) pick_port (contract.sh:72-85) picks a random port in 20000-49999 and checks only that nothing is listening on it, at the start of the script. The server binds long afterwards, after a build, so there is a window. (2) That range overlaps Linux's default ephemeral range (32768-60999), where a port can be held as the source port of an outgoing connection, which lsof -sTCP:LISTEN does not see but a bind refuses. macOS's ephemeral range starts at 49152, which would explain why it is never seen locally.
  • ignore-revoked (the fourth mutant to start) has been the failing one on both pull-request runs where mutants failed. That may be chance.

Proposal

  1. Make it diagnosable first: when a mutant is not killed, print the tail of $OUT/$name.out (the server log) into the job log, and upload target/mutants on failure.
  2. Fix the port choice: take it from below the ephemeral range, or bind port 0 and read the port back, or retry the server start once on address already in use with a fresh port. That is a retry of the start, not of a test; AGENTS.md rule 2 forbids retrying tests, and this must not weaken it.
  3. Separate "did not run" from "survived" in the mutants summary. An exit-4 mutant is not evidence that an assertion is vacuous, and a real survivor must stay unmistakable.

Acceptance criteria

  • A mutant that is not killed shows its server log in the CI job log.
  • The mutants summary reports a mutant whose server never started differently from one that survived, and a planted unkillable mutant still reports SURVIVED (rule 8's control).
  • flake-hunt.yml on the affected cells (worker-cf main, worker-cf bindings), and a repeated run of mutants, show no exit-4 failures over a number of runs agreed in this issue (30, say).
  • If hypothesis 1 or 2 is confirmed or ruled out, the finding is recorded here.

References

#86 (first run), #87, #88; scripts/contract.sh, scripts/mutants.sh; .github/workflows/flake-hunt.yml, contract-cell.yml; AGENTS.md rules 2 and 8.

主要语言
Rust
星标
0
派生
0
平均合并
1 小时 35 分钟
30 天内合并 PR
67

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

aRustyDev/thrust 的其他 Issue

查看 aRustyDev/thrust 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。