Add an optional run outcome to `results`
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 18/100
- Issue 类型
- 功能
- 描述清晰度
- 需要澄清
- 活跃度
- 活跃
- 技术栈
- json
- 领域
- tooling
调研方向
This is an open design question about the CTRF schema, not a code change, and the payload names no repository files. Read #65 and #88 first, along with the linked precedent for Jest, TRX, Cucumber, node:test and Playwright. Done means the maintainers settle the shape (boolean or enum), the consistency rules and the merge rule, and only then is a schema change proposed.
由索引模型根据 Issue 内容生成。
描述
Follow-up to #65, from this comment. #88 adds results.errors and notes that "overall-run-outcome questions remain open." This issue is for that question and should not block 0.3.0.
Problem
With results.errors, a report can describe a failed run whose tests all passed, but it still doesn't carry a verdict. Each consumer has to derive one from summary.failed and results.errors, and the framework's own verdict is lost. Playwright, for example, distinguishes failed, timedout, and interrupted.
Precedent
When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:
- Jest:
successis false whennumRuntimeErrorTestSuites > 0. - VSTest TRX:
ResultSummary/@outcome="Failed"even when every test passed, with the run-level counters left alone. - Cucumber Messages:
TestRunFinished.successis required, along with anexceptionfield. - node:test:
test:summary.successis set false for uncaught errors outside tests, without changingfailed(harness.js). - pytest-json-report: top-level
exitcode. - Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.
The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.
#65 suggested results.environment.healthy as a stand-in, but no consumer I checked reads it. ctrf-io/github-test-reporter gates only on summary.failed.
Questions to settle
- Shape. A boolean like Jest's and Cucumber's
success, or a status enum? An enum could reuse CTRF's test statuses, or add run-specific values such astimedoutandinterrupted. ArawStatus-style field could keep the framework's own value, asresults.tests[].rawStatusdoes for tests. - Consistency. Should a producer that sets a failing outcome also emit a
results.errorsentry (see the "dependable signal" issue)? Should a passing outcome be allowed alongside a non-emptyresults.errorsorsummary.failed > 0? - Merging. Playwright's
merge-reportsmerges run status worst-first: failed, then timedout, then interrupted, then passed. CTRF's merge rule could do the same. - Consumers. When present, consumers like github-test-reporter could gate on the outcome directly, instead of deriving one.
- 主要语言
- 没有语言数据
- 星标
- 103
- 派生
- 4
- 平均合并
- 1 小时 3 分钟
- 30 天内合并 PR
- 2
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
ctrf-io/ctrf 的其他 Issue
-
难度 1/5 1 小时以内 新手友好度 76/100
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 25/100
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 15/100
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 20/100
维护者通常 1 天内回复
-
Proposal: Add optional `threadId` to `retryAttempts` entries可能已有人在做 @alexshamrai 于 12 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 68/100
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 78/100
eliasstravik/herdr-projects#787 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
code-yeongyu/oh-my-openagent#12784 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
-
难度 2/5 1-3 小时 新手友好度 82/100
-
难度 2/5 1-3 小时 新手友好度 71/100
btclib-org/btclib#2964 ·
维护者通常 1 天内回复