fix(ci): contract cells and the mutants job fail intermittently with exit 4 (the server under test never comes up)

Abierto
  • ¿Algo no está bien?
#93 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
github-actions, rust, shell
Área
ci-cd, devops, testing

Línea de trabajo

Start with scripts/contract.sh around pick_port and the exit-4 paths, then read scripts/mutants.sh around mutant output and summary handling. Check .github/workflows/flake-hunt.yml and contract-cell.yml for the affected jobs, and the referenced AGENTS.md rules 2 and 8. Done means server logs are visible for non-killed mutants, startup failures are distinct from survivors, and repeated runs meet the agreed acceptance criteria.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:process bug prio:p1

Context

scripts/contract.sh exits 4 when the server under test never comes up: "target did not become healthy" or "target process exited early" (contract.sh:221 and :226). Contract cells and the mutants job then fail intermittently. Because gate requires every job, one of these turns the required check red, on main and on pull requests, for a reason that says nothing about the code.

scripts/mutants.sh runs each mutant as contract.sh ... > "$OUT/$name.out" 2>&1 (:53), so the server log that contract.sh prints on exit 4 goes into a file the job log does not show. Its summary then labels any non-kill SURVIVED or did not run (exit=..) (:33), so a server that never started reads like a surviving mutant, which is the finding the job exists to make.

Evidence

Where Job What was seen
push to main, b1c9d1f (run 40258128717) mutants native-memory engine + case-sensitive-search SURVIVED or did not run (exit=4, failed tests=)
#86, first run (42158481836) mutants native-memory main + ignore-revoked ... (exit=4, failed tests=)
#86, first run contract (worker-cf, bindings) Address already in use (127.0.0.1:34322), then target process exited early, exit 4
#88, a throwaway pull request (42164639829) mutants ignore-revoked again, exit=4; the other ten mutants were killed
#88 contract (worker-cf, main) Address already in use (127.0.0.1:43896), exit 4

Of the nine pushes to main from ad07a79 to a206319, four failed (ae7dfa6, c0736a0, 7e4e858, b1c9d1f). For c0736a0, 7e4e858 and b1c9d1f the failing job was mutants ("Every mutant must be KILLED"); I did not inspect ae7dfa6. The last three pushes passed.

It is not a real surviving mutant: rerunning the failed jobs of #86 passed, #87's own run was green, and running the ignore-revoked mutant locally on the same tree gave a healthy server and a kill (49 passed, 5 failed, exit 101).

What is not known

  • Why mutants fails: its log does not show the server's. The worker-cf cells are Address already in use.
  • Two hypotheses, neither tested. (1) pick_port (contract.sh:72-85) picks a random port in 20000-49999 and checks only that nothing is listening on it, at the start of the script. The server binds long afterwards, after a build, so there is a window. (2) That range overlaps Linux's default ephemeral range (32768-60999), where a port can be held as the source port of an outgoing connection, which lsof -sTCP:LISTEN does not see but a bind refuses. macOS's ephemeral range starts at 49152, which would explain why it is never seen locally.
  • ignore-revoked (the fourth mutant to start) has been the failing one on both pull-request runs where mutants failed. That may be chance.

Proposal

  1. Make it diagnosable first: when a mutant is not killed, print the tail of $OUT/$name.out (the server log) into the job log, and upload target/mutants on failure.
  2. Fix the port choice: take it from below the ephemeral range, or bind port 0 and read the port back, or retry the server start once on address already in use with a fresh port. That is a retry of the start, not of a test; AGENTS.md rule 2 forbids retrying tests, and this must not weaken it.
  3. Separate "did not run" from "survived" in the mutants summary. An exit-4 mutant is not evidence that an assertion is vacuous, and a real survivor must stay unmistakable.

Acceptance criteria

  • A mutant that is not killed shows its server log in the CI job log.
  • The mutants summary reports a mutant whose server never started differently from one that survived, and a planted unkillable mutant still reports SURVIVED (rule 8's control).
  • flake-hunt.yml on the affected cells (worker-cf main, worker-cf bindings), and a repeated run of mutants, show no exit-4 failures over a number of runs agreed in this issue (30, say).
  • If hypothesis 1 or 2 is confirmed or ruled out, the finding is recorded here.

References

#86 (first run), #87, #88; scripts/contract.sh, scripts/mutants.sh; .github/workflows/flake-hunt.yml, contract-cell.yml; AGENTS.md rules 2 and 8.

Lenguaje dominante
Rust
Estrellas
0
Forks
0
Merge medio
1 h 35 min
PR fusionados (30 d)
67

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de aRustyDev/thrust

Todos los issues de aRustyDev/thrust

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.