tests/e2e: KEEP=on-failure keeps the VM of a failed run and lists kept VMs

Aperta
#176 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
15/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
bash
Ambito
ci-cd, testing-qa

Direzione di ricerca

Start in tests/e2e/lib/vm.sh, next to vm_running and vm_kill, and in on_exit and write_results in tests/e2e/run.sh, which hold the KEEP and N_FAILED logic. The loader, tests/e2e-harness.sh and --print-config come from #170, which must merge first. Done when keep_vm and kept_vms pass the no-root cases and the e2e-harness step in the lint workflow, and docs/e2e-tests.md is updated.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: ci claude enhancement priority: low size: S

Epic #164 · Milestone Phase 1: Community & CI · Priority low · Size S · Depends on #170


Why

--keep keeps the VM of every run, so you have to choose before you know you need it. Playwright keeps a trace when a test fails; here the VM is the trace.
With KEEP=on-failure as the local default, a failed run leaves its VM ready for ssh, and a passed run cleans up after itself. CI never keeps one.
Kept VMs then add up (RAM and a disk of several GB each), so the harness must point at them and say how to remove them.

What to do

  • KEEP=never|on-failure|always in the loader of #170: on-failure in e2e.cfg, never in its CI branch, E2E_KEEP in the environment. A bad value exits 2 with a message that names KEEP. --help lists it.
    • --keep still works, means always, and wins over the files and E2E_KEEP.
    • The setting replaces the KEEP=0/1 variable of run.sh.
  • keep_vm in tests/e2e/lib/vm.sh, next to vm_running and vm_kill. That file is sourced and has no side effects, so a test can call it; run.sh cannot be sourced. It returns 0 when the VM stays: always yes, never no, on-failure when the run failed by the rule of status in write_results (N_FAILED > 0, or ABORT set: time limit, interrupted).
  • on_exit calls keep_vm once, at its top, or right after close_open_scenario of #172 when that has merged, since that call can record a failed scenario. Whichever PR merges second keeps that order. Both of today's [[ $KEEP -eq 1 ]] tests use that answer: the one before vm_kill and the one that chooses between printing the kept VM and rm -rf "$WORK". run.sh runs with set -u, so a -eq test left on a string KEEP stops on_exit before write_results.
  • A kept VM prints three things:
    • its folder;
    • the SSH command, as today: to 127.0.0.1 on the forwarded port, $P822 with keys/panel when CHANNEL is root, else $P22 with keys/ops;
    • a stop command that also frees the disk: kill $(cat DIR/qemu.pid) && rm -rf DIR, or only rm -rf DIR when the VM is not running.
    • Today these lines come before the summary line (<os>: N passed, ...). Move them after it, and after the MP4 lines of #175 when that has merged, so they are the last lines of the run. Only the printing moves: rm -rf "$WORK" for a VM that is not kept stays before the summary line, as #175 expects. Whichever of the two PRs merges second keeps this order.
    • Write the same lines to $WORK/kept-vm.txt before the summary line, so a Ctrl-C later in on_exit cannot lose them. That file marks a kept folder.
  • kept_vms in lib/vm.sh lists every $CACHE_DIR/runs/*/kept-vm.txt. Use CACHE_DIR: E2E_CACHE may be unset.
    • VM running: print its kept-vm.txt.
    • VM stopped: the folder, its size (du -sh) and rm -rf DIR.
    • "Running" means the PID in qemu.pid is alive and /proc/PID/cmdline holds that folder's disk.qcow2. After a host reboot, an old PID can belong to another process.
    • A folder without kept-vm.txt can be a run in progress: mktemp -d makes it before the VM starts, and QEMU writes qemu.pid itself (-pidfile) as soon as it starts. So it is never listed, and two runs at once, or #177, do not report each other's VMs.
    • For the same reason, folders kept by today's --keep and runs killed before on_exit are not listed either, even with a live QEMU. Examples: the SIGKILL of timeout --kill-after=1m in e2e.yml, or a closed terminal. The docs say how to find them by hand (below).
  • run.sh calls kept_vms before the first scenario, as a warning that does not change the run.
  • No-root test: add these cases to tests/e2e-harness.sh from #170. They source lib/vm.sh and lib/record.sh with temporary WORK, OUT and CACHE_DIR.
    • The lint workflow already runs that file: #170 adds its step.
    • keep_vm for each mode with a passed run, a failed one (N_FAILED=1) and an interrupted one (ABORT=interrupted).
    • The loader cases need the resolved KEEP. run.sh cannot show it today: it has three modes only (run, --list-os, --image-key), and its parsing is top-level code. These cases use run.sh --print-config from #170: it prints each resolved setting as a KEY=value (source) line, such as KEEP=always (--keep), and exits 0 before the QEMU tool check, like --list-os.
    • The three loader cases: --keep wins over E2E_KEEP=never (prints KEEP=always); CI=true gives KEEP=never; E2E_KEEP=sometimes exits 2 with a message that names KEEP (exit 2 alone also means a missing --os or a missing QEMU). The local cases run under env -u CI: GitHub Actions sets CI=true in the lint job too.
    • kept_vms: a kept folder with a live fake VM is listed with its kill command. The fake VM is a background process whose command line holds the folder's disk.qcow2, for example bash -c 'sleep 300; :' qemu "file=$dir/disk.qcow2". A kept folder whose PID is dead is listed as stopped, with its rm -rf. A live fake VM without kept-vm.txt is not listed.
  • docs/e2e-tests.md: the KEEP row of the settings table. The --keep line becomes KEEP: how to log in to a kept VM, how to stop it and free its disk, and the warning at the start of a run.
    • Say which folders the warning does not list (runs in progress, old --keep folders, killed runs) and how to find them: ls ~/.cache/server-prep-e2e/runs/ and pgrep -af 'qemu-system-x86_64 -name e2e-' (QEMU runs with -name e2e-<os>).
  • CHANGELOG.md: a line that says a failed local run now keeps its VM.

Acceptance criteria

  • The no-root cases cover the nine keep_vm cases, the three loader cases (through run.sh --print-config) and the three kept_vms cases.
  • A step of the lint workflow runs tests/e2e-harness.sh, where CI=true, and it passes.
  • tests/e2e/run.sh --os debian-13 --only audit-fresh (it runs boot and audit-fresh only) passes, prints no kept: line, and leaves no new folder in the cache's runs/ and no new QEMU process.
  • E2E_FAIL_SCENARIO=audit-fresh tests/e2e/run.sh --os debian-13 --only audit-fresh (the test hook of #170) exits 1. Its last lines are the kept folder, an SSH command that logs in and a stop command that leaves no QEMU process and no folder.
  • A run started while that VM is kept lists it in the start warning. After a plain kill, it is listed as stopped, with its size.
  • --keep still keeps the VM of a passed run.
  • docs/e2e-tests.md names the folders the warning does not list and the two commands that find them.
  • The e2e workflow passes on this PR, and its harness.log files have no kept: line.
  • ShellCheck stays clean.

Not in this issue

  • The loader, tests/e2e-harness.sh, --print-config and the E2E_FAIL_SCENARIO test hook: #170.
  • The MP4 lines printed after the summary: #175.
  • At the end of a run of several systems, printing every kept VM with kept_vms: #177.
Lingua principale
Shell
Stelle
2
Fork
3
Merge medio
2g 14m
PR unite (30g)
47

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di exceev-consulting/server-prep

Tutte le issue di exceev-consulting/server-prep

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.