Make a non-empty `esults.errors` a dependable signal of an unsuccessful run

未关闭
  • 哪里不对？
#92 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
15/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
json
领域
testing

调研方向

This is a specification change, not a code fix: the proposed SHOULD rule belongs in the results.errors section of the CTRF spec, building on #88's results.errors shape. Read #65, #88 and the linked Jest and xUnit precedents to see how other formats record run-level failures. Done means agreed spec wording with maintainer sign-off, and any reporter or consumer changes are tracked in their own repositories.

由索引模型根据 Issue 内容生成。

描述

Follow-up to #65, from this comment. Builds on #88, which adds results.errors: [{message, trace, extra}].

Problem

#88 says "Consumers MUST account for reported global errors when interpreting the run." Nothing requires a producer emit an error when the run fails. Some frameworks report a failed run with only a status and no error object. Playwright is one: FullResult.status can be failed, timedout, or interrupted even when no test failed. A producer that only copies error objects into results.errors writes a report that looks green, and the consumer MUST has nothing to act on.

The broader test-reporting ecosystem keeps fixing this bug over and over:

  • Playwright's own JUnit reporter has no onError handler, so it reports exit 1 with 0 failures.
  • allure-playwright has onError(): void {}.
  • Vitest JSON, node:test JUnit and cargo2junit also drop the error. Vitest's JUnit reporter fixed exactly this in 2026 (PR #10244).
  • playwright-ctrf-json-reporter#123 works around it today with a synthetic failed test, which #88's "Global errors MUST NOT be counted as test results" would forbid.
Precedent

Formats that keep errors separate from tests still guarantee an entry when the run fails:

Proposal

Add to the results.errors section:

A producer SHOULD emit at least one results.errors entry when the framework reports the run as unsuccessful but no test result failed. If the framework provides no error object, the entry SHOULD describe the reported status (for example, "Run ended with status timedout").

With this rule, a consumer can rely on summary.failed > 0 || results.errors.length > 0 to detect a failed run, without knowing each framework's status vocabulary.

Consumer adoption

None of the CI consumers I checked read run-level error fields:

Having github-test-reporter's exit-on-fail also fail on a non-empty results.errors would put the consumer MUST into practice for the most common CTRF consumer. That could be tracked separately in that repo.

Once this lands, I'm happy to rework playwright-ctrf-json-reporter#123 and https://github.com/ctrf-io/jest-ctrf-json-reporter/pull/95to emit results.errors this way.

主要语言
没有语言数据
星标
103
派生
4
平均合并
1 小时 3 分钟
30 天内合并 PR
2

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

ctrf-io/ctrf 的其他 Issue

查看 ctrf-io/ctrf 的全部 Issue

相似的 Issue

更多 Testing & QA Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。