tests/e2e: the report publishes checked recordings and never cuts them

Aperta
#173 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
javascript, shell
Ambito
devops, testing

Direzione di ricerca

Start with build-report.sh (FILE_RE, is_file, JQ_NORM, add_result) and report.js (FILE_RE, normScen, logLink), then scripts/e2e-pages.sh fetch_results. Run bash tests/e2e-report.sh to see the current want-tree and file-state checks. This depends on #169 and #172 for the recording shape and cast writing, so most of it cannot be finished from this issue alone. Done means the listed fixtures and checks pass and the cast states match the acceptance criteria.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: ci claude enhancement priority: high size: M

Epic #164 · Milestone Phase 1: Community & CI · Priority high · Size M · Depends on #169


Why

build-report.sh publishes only names that pass FILE_RE, and it copies a log over 5 MiB as its last 5 MiB. A .cast cut that way loses its header line and no longer plays.
Casts come from VMs that ran pull request code. pages.yml runs scripts/e2e-pages.sh from main. That script builds the reports of main and of open pull requests from this repository or a trusted author, and hosts their files on the project site.
So jq checks and rewrites each cast before it is published: a clean header, then "o" events only. A cast is published whole or not at all.

What to do

The recording field

  • Use the recording shape #169 sets. In a scenario of results.json, it is "recording": {"cast": "<id>.cast", "mp4": "<id>.mp4"}, and both keys are optional. #172 writes cast and #175 writes mp4. Any other shape is ignored.
  • scenario() in JQ_NORM outputs recording: "<id>.cast" only when .recording.cast is exactly that name for the scenario's own valid id, never for the scenario-<n> fallback. Otherwise it outputs "". It never reads .recording.mp4.
  • JQ_NORM lists the accepted cast names in a new casts array, capped at $max_names. Today names is cut to $max_names after the logs and files, so a cast added at the end could be cut, get no attachments entry and never reach the copy loop. So names becomes casts first, then the logs and files cut to $max_names as today, then uniq_by_key(.). A long list of logs can no longer push a recording out. A recording past the cast cap shows as "not published". casts is deleted along with names.

The file rule

  • Add cast to FILE_RE and is_file (in JQ_DEFS) in build-report.sh, and to FILE_RE in report.js. mp4 and every other binary type stay out.
  • A .cast is accepted only as a scenario's recording. A .cast name in logs or files that is not in casts still reaches the copy loop, which marks it refused and never copies it as a log.

The copy loop and the cast check

  • In add_result, the loop runs rm -f -- "$OUT/$os/$name" for every .cast name before the symlink and missing checks, accepted or refused. Today rm -f -- "$dst" runs only in the regular-file branch, and build_report never clears <os>/. So without this, a symlinked, missing or refused cast keeps a file left by an earlier build into the same --out.
  • Then a name in casts goes through the cast check below instead of cat or tail -c. Logs keep their rule: a log over 5 MiB keeps its last 5 MiB.
  • The cast check is one jq program (JQ_CAST). It reads the file line by line with jq -R and fromjson?:
    • Line 1 must be an object with version 2 and numeric width and height. If not, the cast is refused. The header is rebuilt from these keys only: width clamped to 1 to 500, height to 1 to 200, idle_time_limit to 0.1 to 5 s (2 when absent). env, title and every other key are dropped.
    • Each event that is [time, "o", text], with a number time of 0 or more and a string text, is kept. A time that goes down is raised to the previous one. "i", "r", "m" and every other line shape are dropped.
    • Reading stops at the first line that is not JSON, which happens to a cast cut when its run was killed. The cast is then partial, and it keeps the events before that line.
    • A source over MAX_FILE (5 MiB), more than 100000 events or an output over 5 MiB makes the cast too-big. Nothing is published. It is never cut.
    • A symlink is refused and an absent file is missing, as for logs.
  • The check keeps escape sequences in the text as they are (colours, OSC 8, OSC 52): it checks the JSON shape. What the page draws is up to #174.
  • The checked output is written with jq -c (one JSON value per line, still asciicast v2) to a temporary file, then moved to <os>/<id>.cast only when the cast is ok or partial. Any other state leaves no file, thanks to the rm -f above. In states.tsv, the size is the published file's size, or the source size for too-big.
  • Add a budget, MAX_CASTS = 40 MiB, for all the casts published in one report, counted in the order the results are read. A cast that does not fit is over-budget. Without the budget, recordings alone could push a report past MAX_REPORT (100 MiB) in scripts/e2e-pages.sh, which then skips the whole report.
  • If #169 picks jq-built data files over connect-src 'self', also write each published cast as a data file in <os>/, in the format the ADR sets. It needs its own writer, which builds the whole line in jq: jq -a, </ turned into <\/, and a key built from the checked OS and scenario ids. write_js cannot do it: it writes one plain window.NAME = ...; through sed, so it cannot set one key of a shared object. FILE_RE never allows .js, so no result file can take that name. The data files count toward MAX_CASTS, and the same rm -f rule removes a stale one.

The page (report.js)

  • normScen keeps recording when it passes FILE_RE and is <id>.cast. The scenario card draws its Logs block when the scenario has logs or a recording, and lists the recording with logLink(o, name, "Terminal recording"). Until #174 lands, that link is all the page shows, and the page never reads the cast.
  • logLink:
    • also links partial, with an "ends early" tag;
    • gets a too-big branch, "not published: too big", whose title says that a recording over 5 MiB or 100000 events is dropped whole, never cut;
    • gets an over-budget branch, "not published: report full";
    • adds .cast to the list of types in the refused title.

Pages (scripts/e2e-pages.sh)

  • fetch_results unpacks only the names that pass the report's file rule (results.json passes it). An .mp4 or another binary in an artifact can then no longer use up MAX_OS_BYTES or MAX_FILES before the logs. This adds a copy of FILE_RE to the script.
  • fetch_results never cuts a .cast. A cast bigger than what is left of the budget, or than MAX_FILE (32 MiB), is not kept, and a warning says why. A 6 MiB cast still arrives whole, and the report marks it too-big. Fix the MAX_FILE comment ("the report keeps its last 5 MiB"): it now applies to logs only.

Drift test, fixtures and tests

  • A test in tests/e2e-report.sh fails when the type lists differ between the copies of the rule: FILE_RE and is_file in build-report.sh, FILE_RE and the refused title in report.js, and the new copy in scripts/e2e-pages.sh.
  • tests/e2e/fixtures/failed/ (ubuntu-24.04): add a short valid prep-rerun.cast with SGR colours, a \r progress line and a pause longer than its idle_time_limit, and list it as "recording": {"cast": "prep-rerun.cast"} in results.json. Update the two checks in tests/e2e-report.sh that list this fixture's files:
    • want-tree gets ./ubuntu-24.04/prep-rerun.cast;
    • the "file states" expect_jq gets "prep-rerun.cast"=ok first among the ubuntu-24.04 entries, since casts now come first in names.
  • tests/e2e/fixtures/hostile/: add new scenarios, each with its own cast. Update the existing checks for them.
    • a cast with no header line;
    • "i" and "r" events among "o" events, plus a time that goes down;
    • a cast cut in the middle of its last line;
    • text holding </script>, U+2028 and <img src=x onerror=alert(1)>;
    • an OSC 8 link and an OSC 52 clipboard write;
    • a header with width 99999, height 0, idle_time_limit 3600, env and title;
    • recording.cast values of ../x.cast, of another scenario's cast and of x.mp4, a recording that is a string, and a .cast listed in logs.
  • ESC is written \u001b in the fixture casts, so the files stay ASCII. These are the fixtures #174 runs its parser and page checks on.
  • Made in tests/e2e-report.sh, like the 6 MiB log:
    • a symlinked cast, a 6 MiB cast and a cast with 100001 events;
    • an .mp4 listed as recording.mp4 and in logs;
    • a results folder with nine scenarios whose casts are just under 5 MiB each, to test the budget;
    • a results folder whose logs and files list more than 100 names, plus one valid cast;
    • a second build into the same --out after a valid cast has become a symlink, has been removed, or has moved from recording to logs.
  • tests/e2e-pages.sh needs three new artifacts: one with 61 small .mp4 entries whose names sort before the logs, one with a 6 MiB .cast, and one whose .cast sorts after entries that use up most of MAX_OS_BYTES.
  • docs/e2e-tests.md, "Where to see the results": which recordings the report publishes, what the check removes, the states and the size rules.
  • No CHANGELOG.md line: nothing writes a .cast before #172, which adds the line.

Acceptance criteria

  • One expect_jq check lists the state of every cast:

    • ok: the valid one, the "i" one, the </script> one, the OSC one and the out-of-range header;
    • partial: the cut one;
    • refused: no header, the symlink, and the .cast in logs;
    • too-big: 6 MiB, and 100001 events.

    The ../x.cast, other-scenario, x.mp4 and string recording scenarios have no recording.

  • The existing "file states" check passes with "prep-rerun.cast"=ok first for ubuntu-24.04, and want-tree holds ./ubuntu-24.04/prep-rerun.cast.

  • Every published .cast is asciicast v2. Its header has only version, width, height and idle_time_limit (the out-of-range one comes out as 500, 1 and 5). Then come [time, "o", text] lines whose times never go down. No "i" or "r" event is left.

  • The cut cast keeps the events before its broken line. The OSC cast's text is published byte for byte.

  • In the folder with more than 100 log and file names, the cast is ok and published.

  • After the second build into the same --out, the symlinked, removed and moved casts leave no file in <os>/.

  • No .mp4 is copied, whether listed in logs or as recording.mp4, and fetch_results unpacks none. With 61 .mp4 entries ahead of them, every log of the artifact is still published.

  • A 6 MiB cast fetched by scripts/e2e-pages.sh reaches the report whole and is too-big. A cast that does not fit whole in the rest of the budget is not kept, and the log says why. A 6 MiB log is still cut to its last 5 MiB.

  • In the nine-cast build, the casts past 40 MiB are over-budget and leave no file.

  • If data files are written, each one runs alone under node with {window: w} and sets only its own key. None holds </, a raw U+2028 or U+2029, an "i" event, or a key outside the checked ids.

  • The want-tree check lists every new file, and nothing else is copied. "The same input gives the same data.js" also compares the published casts.

  • Changing one copy of the file rule makes the drift test fail.

  • tests/e2e/preview.sh tests/e2e/fixtures/passed tests/e2e/fixtures/failed builds, and the prep-rerun card shows "Terminal recording" under Logs.

  • bash tests/e2e-report.sh, bash tests/e2e-pages.sh and ShellCheck pass.

Not in this issue

  • The choice between data files and connect-src 'self': #169. The player and the CSP: #174.
  • Writing the casts and recording.cast: #172. This issue works on fixtures and can merge before it.
  • Making the MP4 and writing recording.mp4, and making no MP4 in CI: #175.
Lingua principale
Shell
Stelle
2
Fork
3
Merge medio
2g 14m
PR unite (30g)
47

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di exceev-consulting/server-prep

Tutte le issue di exceev-consulting/server-prep

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.