Make a non-empty `esults.errors` a dependable signal of an unsuccessful run

Abierto
  • ¿Algo no está bien?
#92 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
15/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
json
Área
testing

Línea de trabajo

This is a specification change, not a code fix: the proposed SHOULD rule belongs in the results.errors section of the CTRF spec, building on #88's results.errors shape. Read #65, #88 and the linked Jest and xUnit precedents to see how other formats record run-level failures. Done means agreed spec wording with maintainer sign-off, and any reporter or consumer changes are tracked in their own repositories.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Follow-up to #65, from this comment. Builds on #88, which adds results.errors: [{message, trace, extra}].

Problem

#88 says "Consumers MUST account for reported global errors when interpreting the run." Nothing requires a producer emit an error when the run fails. Some frameworks report a failed run with only a status and no error object. Playwright is one: FullResult.status can be failed, timedout, or interrupted even when no test failed. A producer that only copies error objects into results.errors writes a report that looks green, and the consumer MUST has nothing to act on.

The broader test-reporting ecosystem keeps fixing this bug over and over:

  • Playwright's own JUnit reporter has no onError handler, so it reports exit 1 with 0 failures.
  • allure-playwright has onError(): void {}.
  • Vitest JSON, node:test JUnit and cargo2junit also drop the error. Vitest's JUnit reporter fixed exactly this in 2026 (PR #10244).
  • playwright-ctrf-json-reporter#123 works around it today with a synthetic failed test, which #88's "Global errors MUST NOT be counted as test results" would forbid.
Precedent

Formats that keep errors separate from tests still guarantee an entry when the run fails:

Proposal

Add to the results.errors section:

A producer SHOULD emit at least one results.errors entry when the framework reports the run as unsuccessful but no test result failed. If the framework provides no error object, the entry SHOULD describe the reported status (for example, "Run ended with status timedout").

With this rule, a consumer can rely on summary.failed > 0 || results.errors.length > 0 to detect a failed run, without knowing each framework's status vocabulary.

Consumer adoption

None of the CI consumers I checked read run-level error fields:

Having github-test-reporter's exit-on-fail also fail on a non-empty results.errors would put the consumer MUST into practice for the most common CTRF consumer. That could be tracked separately in that repo.

Once this lands, I'm happy to rework playwright-ctrf-json-reporter#123 and https://github.com/ctrf-io/jest-ctrf-json-reporter/pull/95to emit results.errors this way.

Lenguaje dominante
Sin datos de lenguaje
Estrellas
103
Forks
4
Merge medio
1 h 3 min
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ctrf-io/ctrf

Todos los issues de ctrf-io/ctrf

Issues similares

Más issues de Testing & QA

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.