tests/e2e: local runs turn each kept recording into an MP4

Aperta
#175 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
bash, shell
Ambito
ci-cd, testing

Direzione di ricerca

Start in tests/e2e/run.sh, where the tool check loop and on_exit's summary line are, and in the loader from #170, which must land first. The new file tests/e2e/lib/mp4.sh holds mp4_tools and make_mp4s, and tests/e2e-mp4.sh tests them with fake agg and ffmpeg on a restricted PATH. Done means the no-root test passes in lint.yml, the MP4s appear only for kept casts, and results.json gains recording.mp4 without changing duration_s.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: ci claude enhancement priority: medium size: S

Epic #164 · Milestone Phase 1: Community & CI · Priority medium · Size S · Depends on #170, #172


Why

A .cast needs a player. An MP4 opens in any video player and can be attached to an issue, so a local failure can be shown to someone who does not run the tests.
MP4 files are binary and come from VMs that may run pull request code, so they stay local: CI makes none. The tools are optional, and a missing or failing tool never changes a scenario's result.

Decide first

  1. MP4 playback (question 4 of #164).
    • Proposed: idle limit 2 s and speed 1x (agg --idle-time-limit 2 --speed 1), the same idle limit as the cast header.
    • Alternative: a faster speed, such as 2x, for shorter videos of long scenarios like prep-run.
    • Reason: the video keeps the pace of the report player and of an operator, so prompts and questions stay readable. The idle limit already cuts the waits, and every video player can speed up.
  2. Installing agg (question 6 of #164).
    • Proposed: a documented download in docs/e2e-tests.md: a pinned release with its SHA-256, checked with sha256sum -c. The harness only looks on PATH.
    • Alternative: the harness downloads the pinned release into $CACHE_DIR/bin (the E2E_CACHE folder) and checks it, like image_get checks the cloud images.
    • Reason: an e2e run never fetches and runs a binary on the host by itself. Only people who want videos need agg, and CI never does.

What to do

  • VIDEO_MP4=auto|yes|no in the loader of #170: auto in e2e.cfg, no in its CI branch. A bad value exits 2.
    • Its environment variable is E2E_VIDEO_MP4, and VIDEO is E2E_VIDEO: the E2E_<SETTING> form of E2E_DEADLINE_MIN for DEADLINE_MIN. If #170 merges with other names, use those here and in the tests.
    • --help lists E2E_VIDEO_MP4 under Environment:.
  • New tests/e2e/lib/mp4.sh, sourced by run.sh with a # shellcheck source= line like the other libs. It holds:
    • mp4_tools: returns 0 when command -v finds both agg and ffmpeg, 1 otherwise. Prints nothing.
    • make_mp4s, below.
  • Tool check in tests/e2e/run.sh, right after the for tool in qemu-system-x86_64 qemu-img xorriso ... loop and before rm -rf "$OUT":
    • yes: when mp4_tools fails, print the install steps and exit 2.
    • auto: when it fails, the MP4 step is off. Right after the summary line, the run prints one line that says so and points to the docs.
    • no, or VIDEO=off: nothing is checked.
  • make_mp4s converts each $OUT/<id>.cast named in $OUT/results.json.
    • Contract with #172: a scenario's recording is an object, {"cast": "<id>.cast"}, and this issue adds "mp4": "<id>.mp4" to it. With a string, the jq patch below exits 5 (Cannot index string with string "mp4").
    • Run the secret check of #172 on the cast first. A cast that fails it gets no agg call and no MP4: a video would show the password as pixels, which no grep finds.
    • Work in tmp="$(mktemp -d "$OUT/.mp4.XXXXXX")", not in $WORK (deleted by then). The next run's rm -rf "$OUT" clears a folder left by a Ctrl-C.
    • agg --idle-time-limit 2 --speed 1 "$OUT/$id.cast" "$tmp/$id.gif", then ffmpeg -loglevel error -y -i "$tmp/$id.gif" -c:v libx264 -pix_fmt yuv420p -movflags +faststart -vf 'scale=trunc(iw/2)*2:trunc(ih/2)*2' "$tmp/$id.mp4", then mv it to $OUT/$id.mp4 and remove $tmp. -c:v libx264 makes an ffmpeg build without x264 fail instead of writing mpeg4. H.264 with yuv420p needs an even width and height.
    • Patch $OUT/results.json with jq (.scenarios |= map(if .id == $id then .recording.mp4 = $f else . end)) into a temp file in $OUT, then mv it over. Do not call write_results again: it rebuilds the file from $WORK/scenarios.jsonl and recomputes duration_s.
    • Return 1 when a conversion failed.
  • Call make_mp4s in on_exit right after the summary line (<os>: N passed, ...). By then vm_kill and rm -rf "$WORK" have run, and the secret check of #172 has scanned $OUT. A Ctrl-C during a conversion then ends on_exit (the INT trap's exit 1) with the VM already stopped and its disk gone from $CACHE_DIR/runs/, unless the VM is kept.
    • Skip it when [[ $ABORT == interrupted ]]. The INT/TERM trap sets that, and its exit 1 still runs on_exit. Do not test [[ -n $ABORT ]]: ABORT is also time limit reached on a run that ended normally.
    • Its lines, where the MP4s are or the auto install hint, come right after the summary line.
    • When a VM is kept, the kept-VM lines that #176 moves after the summary come after the MP4 lines and stay the last lines of the run. Whichever of the two PRs merges second keeps this order.
  • A failed conversion prints a warning with auto. With yes, on_exit calls exit 1 itself as its last command: the status of an EXIT trap's last command does not change the exit code. Scenario results do not change.
  • CI makes none: a step after Scenarios in .github/workflows/e2e.yml, with if: always(), fails the job when e2e-out/ holds an .mp4. Without if: always() it would be skipped when a scenario fails, and those are the only CI runs that keep casts.
  • No-root tests:
    • The VIDEO_MP4 cases of the loader go in the no-root test of #170: default auto, no with CI=true, E2E_VIDEO_MP4 wins over the file, a bad value exits 2.
    • New tests/e2e-mp4.sh. It sources tests/e2e/lib/record.sh and tests/e2e/lib/mp4.sh, and calls make_mp4s on a fake out/<os>/ with .cast files and a results.json whose recording fields have the object shape above. Fake agg and ffmpeg record their arguments and write their output file. Cases: good fakes, a failing fake, a cast that holds $E2E_ADMIN_PASSWORD, and mp4_tools without agg.
    • Fakes first on PATH cannot hide a real agg in ~/.local/bin or ffmpeg in /usr/bin, next to jq. For the "without agg" cases the test sets PATH to its own folder: links to every command of the real PATH except agg and ffmpeg, plus the fakes. tests/exit-status.sh meets the same problem with tailscale and overrides have, but run.sh calls command -v, so only PATH works here.
    • The same file runs run.sh --os debian-13 --out <tmp> with E2E_VIDEO_MP4=yes, E2E_VIDEO=on, that PATH, and fake qemu-system-x86_64, qemu-img and xorriso. E2E_VIDEO=on keeps a developer's e2e.local.cfg with VIDEO=off from skipping the check.
    • A new step in .github/workflows/lint.yml, next to E2E report builder, runs bash tests/e2e-mp4.sh.
    • CONTRIBUTING.md, "Before you open a pull request": add bash tests/e2e-mp4.sh under bash tests/e2e-report.sh, with a one-line comment, and the same line to the ## Commands block of AGENTS.md.
  • docs/e2e-tests.md, under ## Run it on your machine: installing agg (pinned release, SHA-256 check, into ~/.local/bin) and ffmpeg (sudo apt install ffmpeg), the VIDEO_MP4 row of the settings table, where the MP4s are, and how to attach one to an issue.
  • CHANGELOG.md: one line under ## [Unreleased].

Acceptance criteria

  • tests/e2e-mp4.sh passes in the lint workflow and is listed in CONTRIBUTING.md.
  • With good fakes: one .mp4 per kept cast, agg got --idle-time-limit 2 --speed 1, ffmpeg got -c:v libx264, no GIF or .mp4.* folder is left, and results.json has recording.mp4 next to recording.cast, with an unchanged duration_s.
  • With a failing fake: a warning, make_mp4s returns 1, no half .mp4, and results.json is still valid JSON with the same scenario statuses.
  • A cast that holds $E2E_ADMIN_PASSWORD gets no .mp4, and agg is not called for it.
  • mp4_tools returns 1 without agg, even on a host with agg and ffmpeg installed. E2E_VIDEO_MP4=yes without agg on PATH exits 2 with the agg/ffmpeg message, and the --out folder has no debian-13/.
  • Locally, E2E_VIDEO_MP4=auto tests/e2e/run.sh --os debian-13 --only audit-fresh with agg off PATH prints the install hint after the summary line and exits 0.
  • With CI=true, the loader gives VIDEO_MP4=no. No code in tests/e2e/ downloads agg or ffmpeg, or puts an executable in $E2E_CACHE.
  • Locally, with agg and ffmpeg installed, tests/e2e/run.sh with no flag reads e2e.cfg (debian-13) and leaves a .cast and an .mp4 for every passed scenario. Then E2E_FAIL_SCENARIO=audit-fresh tests/e2e/run.sh (the test hook of #170) leaves both for every passed scenario and for the failed audit-fresh.
  • ffprobe shows h264 and yuv420p for each MP4, and prep-interactive.mp4 plays in Firefox and Chrome.
  • A Ctrl-C during the MP4 step of a passed run leaves no run folder in $CACHE_DIR/runs/, no half $OUT/<id>.mp4, and a valid results.json. A .mp4.XXXXXX/ folder may remain until the next run.
  • The no-MP4 step in e2e.yml has if: always() and passes on every system in this PR's e2e run. ShellCheck stays clean.

Not in this issue

  • Writing the casts, the recording field, the VIDEO modes and the secret check: #172.
  • The loader, e2e.cfg, its CI branch, the E2E_<SETTING> names and E2E_FAIL_SCENARIO: #170.
  • Keeping a VM and the kept-VM lines: #176.
  • Running without --os on several systems: #177.
  • Keeping MP4 files out of the report and off the Pages site: #173.
Lingua principale
Shell
Stelle
2
Fork
3
Merge medio
2g 14m
PR unite (30g)
47

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di exceev-consulting/server-prep

Tutte le issue di exceev-consulting/server-prep

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.