Make a non-empty `esults.errors` a dependable signal of an unsuccessful run

Aperta
  • Qualcosa non va?
#92 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
json
Ambito
testing

Direzione di ricerca

This is a specification change, not a code fix: the proposed SHOULD rule belongs in the results.errors section of the CTRF spec, building on #88's results.errors shape. Read #65, #88 and the linked Jest and xUnit precedents to see how other formats record run-level failures. Done means agreed spec wording with maintainer sign-off, and any reporter or consumer changes are tracked in their own repositories.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Follow-up to #65, from this comment. Builds on #88, which adds results.errors: [{message, trace, extra}].

Problem

#88 says "Consumers MUST account for reported global errors when interpreting the run." Nothing requires a producer emit an error when the run fails. Some frameworks report a failed run with only a status and no error object. Playwright is one: FullResult.status can be failed, timedout, or interrupted even when no test failed. A producer that only copies error objects into results.errors writes a report that looks green, and the consumer MUST has nothing to act on.

The broader test-reporting ecosystem keeps fixing this bug over and over:

  • Playwright's own JUnit reporter has no onError handler, so it reports exit 1 with 0 failures.
  • allure-playwright has onError(): void {}.
  • Vitest JSON, node:test JUnit and cargo2junit also drop the error. Vitest's JUnit reporter fixed exactly this in 2026 (PR #10244).
  • playwright-ctrf-json-reporter#123 works around it today with a synthetic failed test, which #88's "Global errors MUST NOT be counted as test results" would forbid.
Precedent

Formats that keep errors separate from tests still guarantee an entry when the run fails:

Proposal

Add to the results.errors section:

A producer SHOULD emit at least one results.errors entry when the framework reports the run as unsuccessful but no test result failed. If the framework provides no error object, the entry SHOULD describe the reported status (for example, "Run ended with status timedout").

With this rule, a consumer can rely on summary.failed > 0 || results.errors.length > 0 to detect a failed run, without knowing each framework's status vocabulary.

Consumer adoption

None of the CI consumers I checked read run-level error fields:

Having github-test-reporter's exit-on-fail also fail on a non-empty results.errors would put the consumer MUST into practice for the most common CTRF consumer. That could be tracked separately in that repo.

Once this lands, I'm happy to rework playwright-ctrf-json-reporter#123 and https://github.com/ctrf-io/jest-ctrf-json-reporter/pull/95to emit results.errors this way.

Lingua principale
Nessun dato sulla lingua
Stelle
103
Fork
4
Merge medio
1h 3m
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ctrf-io/ctrf

Tutte le issue di ctrf-io/ctrf

Issue simili

Altre issue su Testing & QA

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.