[quality] e2e coverage failure reports say 'run null; status unknown' for every error raised after the manifest is read
维护者通常 1 天内回复
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 78/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- github-actions, javascript, node.js
调研方向
从 tests/tools/e2e-coverage-report.mjs 中的 catch 块以及 tests/e2e-coverage-report.test.mjs 中的 CLI 用例开始;运行 npm run test:unit:coverage 查看现有的覆盖率测试。为可读取的成功和失败 manifest 添加会触发普通 Error 的用例,并添加 manifest 缺失或不可读取的用例。完成标准是:失败报告在有 manifest 时保留其 runId/runStatus,保留 failed/cancelled 状态,并且在无法读取 manifest 时仍使用 null 字段。
由索引模型根据 Issue 内容生成。
描述
Finding
When tests/tools/e2e-coverage-report.mjs fails for any reason after the run
manifest has been read and validated, the failure report it publishes claims it
does not know which run it was rendering — even though it does.
main() builds its fallback report from fields that only CoverageReportError
carries (tests/tools/e2e-coverage-report.mjs:688-696):
status:
error.runStatus === 'failed' || error.runStatus === 'cancelled'
? error.runStatus
: 'tooling-error',
runId: error.runId ?? null,
runStatus: error.runStatus ?? null,
Only four throw sites construct a CoverageReportError, and all four are the
run-status/capture guards near the top of collectE2ECoverage
(lines 461, 467, 484). Every later failure is a plain Error:
coverage run contains incomplete temporary artifacts(line 474)unexpected coverage artifact kind in <entry>(line 490)coverage artifact <entry> belongs to another run(line 493)No src/** coverage was attributable(line 619)- every per-script conversion failure —
source map escapes build directory
(162, 166),generated source has no sourceMappingURL comment(178),
generated source hash mismatch(369), and the map-resolution throws at
211, 222, 229, 260, 344, 351, 361.
For all of those, error.runId and error.runStatus are undefined, so
renderE2ECoverageReport (line 626) emits:
E2E coverage: tooling-error (run null; status unknown)
Why it matters
.github/workflows/ci.yml:236-249 cats coverage/e2e/report.txt straight into
$GITHUB_STEP_SUMMARY, and that header line is the whole diagnostic. A reader
of a red End-to-end coverage job cannot tell from it whether
- the Playwright run itself failed and the reporter is faithfully reporting a
sealed-failedrun, or - the Playwright run passed and only the reporting step broke.
Those two have opposite remedies, and the artifact erases the distinction at
exactly the moment it is needed. The same null lands in report.json, so
anything consuming the JSON inherits it.
There is a second, smaller consequence: a run sealed failed or cancelled
that then trips one of the plain-Error sites is reported with
status: 'tooling-error' rather than failed/cancelled, because the ternary
above also reads error.runStatus. The report blames the tooling for a test
failure.
Reproduction
At 900592b, render any sealed-passed run whose scripts cannot be attributed
to the local build/ — e.g. a published e2e-coverage artifact unpacked next
to a different build:
$ node tests/tools/e2e-coverage-run.mjs seal --dir "$RUN_DIR" --status passed
{"schemaVersion":1,...,"runId":"local1","status":"passed",...}
$ npm run report:e2e:coverage -- --input "$RUN_DIR" --build build --text out.txt
$ head -1 out.txt
E2E coverage: tooling-error (run null; status unknown)
The manifest on disk says "runId": "local1", "status": "passed". The report
says null / unknown.
Evidence and provenance
- Unit:
npm run test:unit:coverage(TZ=UTC node tests/tools/coverage-report.mjs,
node v26.10.0), local, revision900592b—tests/tools/e2e-coverage-report.mjs
100.00% lines / 90.68% regions. Themain()catch block is executed by the
existing CLI tests, but no test asserts therunId/runStatusit writes for
a non-CoverageReportErrorfailure, which is why the defect is invisible to
the suite. - Reproduced locally twice at
900592bas shown above, once against a freshly
sealed local run and once against thee2e-coverageartifact of CI run
37164361552. - This is a reporting-fidelity defect, not a coverage gap: no claim is made
here about any source path's end-to-end coverage.
Recommendation
Fall back to the run manifest in main()'s catch, so the published failure
report names the run whenever the manifest is readable:
} catch (error) {
// Only the run-status guards raise CoverageReportError; every later
// failure is a plain Error, so re-read the manifest rather than publish a
// report that cannot say whether the run passed or the tooling broke.
const run = await readCoverageRun(resolve(options.input)).catch(() => null);
const runStatus = error.runStatus ?? run?.status ?? null;
const failure = {
schemaVersion: 2,
kind: ERROR_KIND,
status:
runStatus === 'failed' || runStatus === 'cancelled'
? runStatus
: 'tooling-error',
runId: error.runId ?? run?.runId ?? null,
runStatus,
...
readCoverageRun is already imported at line 14, and the .catch(() => null)
keeps an unreadable or absent manifest behaving exactly as it does today.
Cover it in tests/e2e-coverage-report.test.mjs with two CLI cases:
- a sealed-
passedrun that trips a plain-Errorsite, asserting the written
report carries thatrunIdandrunStatus: 'passed'with
status: 'tooling-error'; - a sealed-
failedrun that trips the same site, assertingstatusis
'failed'rather than'tooling-error'.
Coordination
The fix lands only in main() (tests/tools/e2e-coverage-report.mjs:674-713).
- #1040 edits the same file at
convertScriptand insidecollectE2ECoverage(script-root resolution); it
does not touchmain()or anythrowsite. - #1051 edits
getRegionCoverage
and the union; it does not touchmain().
No open PR claims this function, so the change is disjoint from both.
Completion criteria
- A failure report for a readable manifest carries the run's real
runIdandrunStatus - A run sealed
failed/cancelledis reported with that status, nottooling-error - An unreadable or absent manifest still yields
runId: null/runStatus: null -
tests/e2e-coverage-report.test.mjscovers all three
Priority
- Impact: medium (CI's only published e2e diagnostic misattributes test failures to the tooling)
- Effort: low
🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: 900592b
— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88
- 主要语言
- JavaScript
- 星标
- 0
- 派生
- 2
- 平均合并
- 21 小时 59 分钟
- 30 天内合并 PR
- 431
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
cncf/endusers 的其他 Issue
-
agent/scanner hive/hosted-available-lke648397-260827-5n31 quality testing
难度 2/5 1-3 小时 新手友好度 88/100
维护者通常 1 天内回复
-
enhancement security
难度 5/5 一周以上 新手友好度 35/100
维护者通常 1 天内回复
-
agent/quality hive/hosted-available-lke648397-260827-5n31 quality testing
难度 4/5 3-5 天 新手友好度 42/100
维护者通常 1 天内回复
-
quality testing
难度 4/5 3-5 天 新手友好度 52/100
维护者通常 1 天内回复
-
agent/quality hive/hosted-available-lke648397-260827-5n31 hive/verified-open needs-human quality testing
难度 4/5 3-5 天 新手友好度 35/100
维护者通常 1 天内回复
相似的 Issue
-
Bump Firebase JS SDK (12.19.0 → 13.0.0)可能已有人在做 @SelaseKay 今天认领。 未关闭Needs Attention type: enhancement
难度 2/5 1-3 小时 新手友好度 75/100
invertase/react-native-firebase#9364 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 67/100
tchiotludo/akhq#3307 · 1 个 reaction ·
维护者通常 1 天内回复
-
难度 1/5 1-3 小时 新手友好度 90/100
DietrichGebert/ponytail#1063 ·
维护者通常 3 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
zen-browser/desktop#15809 · 1 个 reaction ·
维护者通常 1 天内回复
-
enhancement
难度 1/5 1 小时以内 新手友好度 90/100