Add an optional run outcome to `results`

Abierto
  • ¿Algo no está bien?
#94 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
18/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
json
Área
tooling

Línea de trabajo

This is an open design question about the CTRF schema, not a code change, and the payload names no repository files. Read #65 and #88 first, along with the linked precedent for Jest, TRX, Cucumber, node:test and Playwright. Done means the maintainers settle the shape (boolean or enum), the consistency rules and the merge rule, and only then is a schema change proposed.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Follow-up to #65, from this comment. #88 adds results.errors and notes that "overall-run-outcome questions remain open." This issue is for that question and should not block 0.3.0.

Problem

With results.errors, a report can describe a failed run whose tests all passed, but it still doesn't carry a verdict. Each consumer has to derive one from summary.failed and results.errors, and the framework's own verdict is lost. Playwright, for example, distinguishes failed, timedout, and interrupted.

Precedent

When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:

  • Jest: success is false when numRuntimeErrorTestSuites > 0.
  • VSTest TRX: ResultSummary/@outcome="Failed" even when every test passed, with the run-level counters left alone.
  • Cucumber Messages: TestRunFinished.success is required, along with an exception field.
  • node:test: test:summary.success is set false for uncaught errors outside tests, without changing failed (harness.js).
  • pytest-json-report: top-level exitcode.
  • Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.

The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.

#65 suggested results.environment.healthy as a stand-in, but no consumer I checked reads it. ctrf-io/github-test-reporter gates only on summary.failed.

Questions to settle
  • Shape. A boolean like Jest's and Cucumber's success, or a status enum? An enum could reuse CTRF's test statuses, or add run-specific values such as timedout and interrupted. A rawStatus-style field could keep the framework's own value, as results.tests[].rawStatus does for tests.
  • Consistency. Should a producer that sets a failing outcome also emit a results.errors entry (see the "dependable signal" issue)? Should a passing outcome be allowed alongside a non-empty results.errors or summary.failed > 0?
  • Merging. Playwright's merge-reports merges run status worst-first: failed, then timedout, then interrupted, then passed. CTRF's merge rule could do the same.
  • Consumers. When present, consumers like github-test-reporter could gate on the outcome directly, instead of deriving one.
Lenguaje dominante
Sin datos de lenguaje
Estrellas
103
Forks
4
Merge medio
1 h 3 min
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ctrf-io/ctrf

Todos los issues de ctrf-io/ctrf

Issues similares

Más issues de DevTools

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.