Add an optional run outcome to `results`

Ouverte
  • Quelque chose ne va pas ?
#94 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
18/100
Type d'issue
Fonctionnalité
Clarté
À clarifier
Activité
Active
Stack technique
json
Domaine
tooling

Piste de recherche

This is an open design question about the CTRF schema, not a code change, and the payload names no repository files. Read #65 and #88 first, along with the linked precedent for Jest, TRX, Cucumber, node:test and Playwright. Done means the maintainers settle the shape (boolean or enum), the consistency rules and the merge rule, and only then is a schema change proposed.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Follow-up to #65, from this comment. #88 adds results.errors and notes that "overall-run-outcome questions remain open." This issue is for that question and should not block 0.3.0.

Problem

With results.errors, a report can describe a failed run whose tests all passed, but it still doesn't carry a verdict. Each consumer has to derive one from summary.failed and results.errors, and the framework's own verdict is lost. Playwright, for example, distinguishes failed, timedout, and interrupted.

Precedent

When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:

  • Jest: success is false when numRuntimeErrorTestSuites > 0.
  • VSTest TRX: ResultSummary/@outcome="Failed" even when every test passed, with the run-level counters left alone.
  • Cucumber Messages: TestRunFinished.success is required, along with an exception field.
  • node:test: test:summary.success is set false for uncaught errors outside tests, without changing failed (harness.js).
  • pytest-json-report: top-level exitcode.
  • Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.

The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.

#65 suggested results.environment.healthy as a stand-in, but no consumer I checked reads it. ctrf-io/github-test-reporter gates only on summary.failed.

Questions to settle
  • Shape. A boolean like Jest's and Cucumber's success, or a status enum? An enum could reuse CTRF's test statuses, or add run-specific values such as timedout and interrupted. A rawStatus-style field could keep the framework's own value, as results.tests[].rawStatus does for tests.
  • Consistency. Should a producer that sets a failing outcome also emit a results.errors entry (see the "dependable signal" issue)? Should a passing outcome be allowed alongside a non-empty results.errors or summary.failed > 0?
  • Merging. Playwright's merge-reports merges run status worst-first: failed, then timedout, then interrupted, then passed. CTRF's merge rule could do the same.
  • Consumers. When present, consumers like github-test-reporter could gate on the outcome directly, instead of deriving one.
Langage dominant
Aucune donnée de langage
Étoiles
103
Forks
4
Merge moyen
1 h 3 min
PR mergées (30 j)
2

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de ctrf-io/ctrf

Toutes les issues de ctrf-io/ctrf

Issues similaires

Plus d'issues DevTools

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.