Add an optional run outcome to `results`

Aperta
  • Qualcosa non va?
#94 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
18/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
json
Ambito
tooling

Direzione di ricerca

This is an open design question about the CTRF schema, not a code change, and the payload names no repository files. Read #65 and #88 first, along with the linked precedent for Jest, TRX, Cucumber, node:test and Playwright. Done means the maintainers settle the shape (boolean or enum), the consistency rules and the merge rule, and only then is a schema change proposed.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Follow-up to #65, from this comment. #88 adds results.errors and notes that "overall-run-outcome questions remain open." This issue is for that question and should not block 0.3.0.

Problem

With results.errors, a report can describe a failed run whose tests all passed, but it still doesn't carry a verdict. Each consumer has to derive one from summary.failed and results.errors, and the framework's own verdict is lost. Playwright, for example, distinguishes failed, timedout, and interrupted.

Precedent

When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:

  • Jest: success is false when numRuntimeErrorTestSuites > 0.
  • VSTest TRX: ResultSummary/@outcome="Failed" even when every test passed, with the run-level counters left alone.
  • Cucumber Messages: TestRunFinished.success is required, along with an exception field.
  • node:test: test:summary.success is set false for uncaught errors outside tests, without changing failed (harness.js).
  • pytest-json-report: top-level exitcode.
  • Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.

The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.

#65 suggested results.environment.healthy as a stand-in, but no consumer I checked reads it. ctrf-io/github-test-reporter gates only on summary.failed.

Questions to settle
  • Shape. A boolean like Jest's and Cucumber's success, or a status enum? An enum could reuse CTRF's test statuses, or add run-specific values such as timedout and interrupted. A rawStatus-style field could keep the framework's own value, as results.tests[].rawStatus does for tests.
  • Consistency. Should a producer that sets a failing outcome also emit a results.errors entry (see the "dependable signal" issue)? Should a passing outcome be allowed alongside a non-empty results.errors or summary.failed > 0?
  • Merging. Playwright's merge-reports merges run status worst-first: failed, then timedout, then interrupted, then passed. CTRF's merge rule could do the same.
  • Consumers. When present, consumers like github-test-reporter could gate on the outcome directly, instead of deriving one.
Lingua principale
Nessun dato sulla lingua
Stelle
103
Fork
4
Merge medio
1h 3m
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ctrf-io/ctrf

Tutte le issue di ctrf-io/ctrf

Issue simili

Altre issue su DevTools

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.