Make a non-empty `esults.errors` a dependable signal of an unsuccessful run

Aberta
  • Algo errado?
#92 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
15/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
json
Domínio
testing

Direção de pesquisa

This is a specification change, not a code fix: the proposed SHOULD rule belongs in the results.errors section of the CTRF spec, building on #88's results.errors shape. Read #65, #88 and the linked Jest and xUnit precedents to see how other formats record run-level failures. Done means agreed spec wording with maintainer sign-off, and any reporter or consumer changes are tracked in their own repositories.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Follow-up to #65, from this comment. Builds on #88, which adds results.errors: [{message, trace, extra}].

Problem

#88 says "Consumers MUST account for reported global errors when interpreting the run." Nothing requires a producer emit an error when the run fails. Some frameworks report a failed run with only a status and no error object. Playwright is one: FullResult.status can be failed, timedout, or interrupted even when no test failed. A producer that only copies error objects into results.errors writes a report that looks green, and the consumer MUST has nothing to act on.

The broader test-reporting ecosystem keeps fixing this bug over and over:

  • Playwright's own JUnit reporter has no onError handler, so it reports exit 1 with 0 failures.
  • allure-playwright has onError(): void {}.
  • Vitest JSON, node:test JUnit and cargo2junit also drop the error. Vitest's JUnit reporter fixed exactly this in 2026 (PR #10244).
  • playwright-ctrf-json-reporter#123 works around it today with a synthetic failed test, which #88's "Global errors MUST NOT be counted as test results" would forbid.
Precedent

Formats that keep errors separate from tests still guarantee an entry when the run fails:

Proposal

Add to the results.errors section:

A producer SHOULD emit at least one results.errors entry when the framework reports the run as unsuccessful but no test result failed. If the framework provides no error object, the entry SHOULD describe the reported status (for example, "Run ended with status timedout").

With this rule, a consumer can rely on summary.failed > 0 || results.errors.length > 0 to detect a failed run, without knowing each framework's status vocabulary.

Consumer adoption

None of the CI consumers I checked read run-level error fields:

Having github-test-reporter's exit-on-fail also fail on a non-empty results.errors would put the consumer MUST into practice for the most common CTRF consumer. That could be tracked separately in that repo.

Once this lands, I'm happy to rework playwright-ctrf-json-reporter#123 and https://github.com/ctrf-io/jest-ctrf-json-reporter/pull/95to emit results.errors this way.

Linguagem predominante
Sem dados de linguagem
Estrelas
104
Forks
4
Merge médio
1h 3min
PRs com merge (30d)
2

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de ctrf-io/ctrf

Todas as issues de ctrf-io/ctrf

Issues semelhantes

Mais issues de Testing & QA

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.