fix(ci): contract cells and the mutants job fail intermittently with exit 4 (the server under test never comes up)

Aperta
  • Qualcosa non va?
#93 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
github-actions, rust, shell
Ambito
ci-cd, devops, testing

Direzione di ricerca

Start with scripts/contract.sh around pick_port and the exit-4 paths, then read scripts/mutants.sh around mutant output and summary handling. Check .github/workflows/flake-hunt.yml and contract-cell.yml for the affected jobs, and the referenced AGENTS.md rules 2 and 8. Done means server logs are visible for non-killed mutants, startup failures are distinct from survivors, and repeated runs meet the agreed acceptance criteria.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:process bug prio:p1

Context

scripts/contract.sh exits 4 when the server under test never comes up: "target did not become healthy" or "target process exited early" (contract.sh:221 and :226). Contract cells and the mutants job then fail intermittently. Because gate requires every job, one of these turns the required check red, on main and on pull requests, for a reason that says nothing about the code.

scripts/mutants.sh runs each mutant as contract.sh ... > "$OUT/$name.out" 2>&1 (:53), so the server log that contract.sh prints on exit 4 goes into a file the job log does not show. Its summary then labels any non-kill SURVIVED or did not run (exit=..) (:33), so a server that never started reads like a surviving mutant, which is the finding the job exists to make.

Evidence

Where Job What was seen
push to main, b1c9d1f (run 40258128717) mutants native-memory engine + case-sensitive-search SURVIVED or did not run (exit=4, failed tests=)
#86, first run (42158481836) mutants native-memory main + ignore-revoked ... (exit=4, failed tests=)
#86, first run contract (worker-cf, bindings) Address already in use (127.0.0.1:34322), then target process exited early, exit 4
#88, a throwaway pull request (42164639829) mutants ignore-revoked again, exit=4; the other ten mutants were killed
#88 contract (worker-cf, main) Address already in use (127.0.0.1:43896), exit 4

Of the nine pushes to main from ad07a79 to a206319, four failed (ae7dfa6, c0736a0, 7e4e858, b1c9d1f). For c0736a0, 7e4e858 and b1c9d1f the failing job was mutants ("Every mutant must be KILLED"); I did not inspect ae7dfa6. The last three pushes passed.

It is not a real surviving mutant: rerunning the failed jobs of #86 passed, #87's own run was green, and running the ignore-revoked mutant locally on the same tree gave a healthy server and a kill (49 passed, 5 failed, exit 101).

What is not known

  • Why mutants fails: its log does not show the server's. The worker-cf cells are Address already in use.
  • Two hypotheses, neither tested. (1) pick_port (contract.sh:72-85) picks a random port in 20000-49999 and checks only that nothing is listening on it, at the start of the script. The server binds long afterwards, after a build, so there is a window. (2) That range overlaps Linux's default ephemeral range (32768-60999), where a port can be held as the source port of an outgoing connection, which lsof -sTCP:LISTEN does not see but a bind refuses. macOS's ephemeral range starts at 49152, which would explain why it is never seen locally.
  • ignore-revoked (the fourth mutant to start) has been the failing one on both pull-request runs where mutants failed. That may be chance.

Proposal

  1. Make it diagnosable first: when a mutant is not killed, print the tail of $OUT/$name.out (the server log) into the job log, and upload target/mutants on failure.
  2. Fix the port choice: take it from below the ephemeral range, or bind port 0 and read the port back, or retry the server start once on address already in use with a fresh port. That is a retry of the start, not of a test; AGENTS.md rule 2 forbids retrying tests, and this must not weaken it.
  3. Separate "did not run" from "survived" in the mutants summary. An exit-4 mutant is not evidence that an assertion is vacuous, and a real survivor must stay unmistakable.

Acceptance criteria

  • A mutant that is not killed shows its server log in the CI job log.
  • The mutants summary reports a mutant whose server never started differently from one that survived, and a planted unkillable mutant still reports SURVIVED (rule 8's control).
  • flake-hunt.yml on the affected cells (worker-cf main, worker-cf bindings), and a repeated run of mutants, show no exit-4 failures over a number of runs agreed in this issue (30, say).
  • If hypothesis 1 or 2 is confirmed or ruled out, the finding is recorded here.

References

#86 (first run), #87, #88; scripts/contract.sh, scripts/mutants.sh; .github/workflows/flake-hunt.yml, contract-cell.yml; AGENTS.md rules 2 and 8.

Lingua principale
Rust
Stelle
0
Fork
0
Merge medio
1h 35m
PR unite (30g)
67

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di aRustyDev/thrust

Tutte le issue di aRustyDev/thrust

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.