fix(ci): contract cells and the mutants job fail intermittently with exit 4 (the server under test never comes up)

未關閉
  • 哪裡不對？
#93 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
4/5
預估耗時
3-5 天
新手友好度
38/100
Issue 類型
缺陷
描述清晰度
描述清楚
活躍度
活躍
技術堆疊
github-actions, rust, shell
領域
ci-cd, devops, testing

研究方向

Start with scripts/contract.sh around pick_port and the exit-4 paths, then read scripts/mutants.sh around mutant output and summary handling. Check .github/workflows/flake-hunt.yml and contract-cell.yml for the affected jobs, and the referenced AGENTS.md rules 2 and 8. Done means server logs are visible for non-killed mutants, startup failures are distinct from survivors, and repeated runs meet the agreed acceptance criteria.

由索引模型根據 Issue 內容生成。

描述

area:process bug prio:p1

Context

scripts/contract.sh exits 4 when the server under test never comes up: "target did not become healthy" or "target process exited early" (contract.sh:221 and :226). Contract cells and the mutants job then fail intermittently. Because gate requires every job, one of these turns the required check red, on main and on pull requests, for a reason that says nothing about the code.

scripts/mutants.sh runs each mutant as contract.sh ... > "$OUT/$name.out" 2>&1 (:53), so the server log that contract.sh prints on exit 4 goes into a file the job log does not show. Its summary then labels any non-kill SURVIVED or did not run (exit=..) (:33), so a server that never started reads like a surviving mutant, which is the finding the job exists to make.

Evidence

Where Job What was seen
push to main, b1c9d1f (run 40258128717) mutants native-memory engine + case-sensitive-search SURVIVED or did not run (exit=4, failed tests=)
#86, first run (42158481836) mutants native-memory main + ignore-revoked ... (exit=4, failed tests=)
#86, first run contract (worker-cf, bindings) Address already in use (127.0.0.1:34322), then target process exited early, exit 4
#88, a throwaway pull request (42164639829) mutants ignore-revoked again, exit=4; the other ten mutants were killed
#88 contract (worker-cf, main) Address already in use (127.0.0.1:43896), exit 4

Of the nine pushes to main from ad07a79 to a206319, four failed (ae7dfa6, c0736a0, 7e4e858, b1c9d1f). For c0736a0, 7e4e858 and b1c9d1f the failing job was mutants ("Every mutant must be KILLED"); I did not inspect ae7dfa6. The last three pushes passed.

It is not a real surviving mutant: rerunning the failed jobs of #86 passed, #87's own run was green, and running the ignore-revoked mutant locally on the same tree gave a healthy server and a kill (49 passed, 5 failed, exit 101).

What is not known

  • Why mutants fails: its log does not show the server's. The worker-cf cells are Address already in use.
  • Two hypotheses, neither tested. (1) pick_port (contract.sh:72-85) picks a random port in 20000-49999 and checks only that nothing is listening on it, at the start of the script. The server binds long afterwards, after a build, so there is a window. (2) That range overlaps Linux's default ephemeral range (32768-60999), where a port can be held as the source port of an outgoing connection, which lsof -sTCP:LISTEN does not see but a bind refuses. macOS's ephemeral range starts at 49152, which would explain why it is never seen locally.
  • ignore-revoked (the fourth mutant to start) has been the failing one on both pull-request runs where mutants failed. That may be chance.

Proposal

  1. Make it diagnosable first: when a mutant is not killed, print the tail of $OUT/$name.out (the server log) into the job log, and upload target/mutants on failure.
  2. Fix the port choice: take it from below the ephemeral range, or bind port 0 and read the port back, or retry the server start once on address already in use with a fresh port. That is a retry of the start, not of a test; AGENTS.md rule 2 forbids retrying tests, and this must not weaken it.
  3. Separate "did not run" from "survived" in the mutants summary. An exit-4 mutant is not evidence that an assertion is vacuous, and a real survivor must stay unmistakable.

Acceptance criteria

  • A mutant that is not killed shows its server log in the CI job log.
  • The mutants summary reports a mutant whose server never started differently from one that survived, and a planted unkillable mutant still reports SURVIVED (rule 8's control).
  • flake-hunt.yml on the affected cells (worker-cf main, worker-cf bindings), and a repeated run of mutants, show no exit-4 failures over a number of runs agreed in this issue (30, say).
  • If hypothesis 1 or 2 is confirmed or ruled out, the finding is recorded here.

References

#86 (first run), #87, #88; scripts/contract.sh, scripts/mutants.sh; .github/workflows/flake-hunt.yml, contract-cell.yml; AGENTS.md rules 2 and 8.

主要語言
Rust
星號
0
分支
0
平均合併
1 小時 35 分鐘
30 天內合併 PR
67

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

aRustyDev/thrust 的其他 Issue

查看 aRustyDev/thrust 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。