Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[quality] e2e coverage gate's measurement record is stale and nothing guards the suite it was measured against

Closed
  • Is something wrong?
#1,447 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #1449 by @hivecommons-hive — merged

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
66/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript, nodejs
Domain
testing

Research direction

Start in tests/e2e-coverage-gate.test.mjs, where the measurement table and the MEASURED_ON_NODE_MAJOR guard live, and read CONTRIBUTING.md's coverage recipe. Replace the stale figures with the re-measured ones in the issue, then add a MEASURED_AT_SPEC_COUNT constant checked against readdirSync('tests/e2e'). Done when both failure arms go red against fixture directories and the existing suite still passes.

Written by the indexing model from the issue text.

Description

agent/quality hive/covered-by-pr hive/hosted-available-lke648397-260827-5n31 quality testing

Finding

tests/e2e-coverage-gate.test.mjs carries the measurement record the e2e region
floors rest on — a two-column table of per-file and aggregate region counts on
the Node major ci.yml pins versus a newer one. The file states its own purpose
for it:

A contributor who follows CONTRIBUTING.md's local recipe on a newer Node reads
the second column, concludes the floors have 9 points of slack, ratchets them,
and turns the required job red.

The table is stale, and nothing guards it. It describes "the same 344-test
suite". At eff33b5 the suite is 40 spec files / 356 tests under
E2E_COVERAGE=1. Every region count in it has moved.

Re-measured both columns at eff33b5 with CONTRIBUTING.md's recipe (full
build:e2e:coverage + test:e2e:coverage + report:e2e:coverage, 356/356
passed on each runtime):

file recorded 22.23.3 measured 22.21.1 recorded 26.10.0 measured 26.10.0
MemberDirectory/DirectoryFreshness.js 11/15 73.33% 11/15 73.33% 9/10 90.00% 11/12 91.67%
PeopleFreshness/index.js 2/3 66.67% 2/3 66.67% 3/4 75.00% 2/3 66.67%
CNCFProjectCard/index.js 26/26 100.00% 26/26 100.00% 26/26 100.00% 26/26 100.00%
src files (aggregate) 645/663 97.29% 651/671 97.02% 647/662 97.73% 650/667 97.45%

The parenthetical "the same sources measured 9/13 and 649/667 on 22.21.1" also
reproduces as 11/15 and 651/671 on that exact patch.

The staleness points the dangerous way. The node-26 column reads like three
regions of slack above --check-source-regions 96 (97% of 667 = 647, with 650
covered). The authoritative node-22 figure is 651/671: 97% of 671 = 650.87, so a
ratchet to 97 would demand 651 of 671 exactly — a gate with no slack at all.
96 remains the only workable value, which is the opposite of what the recorded
table suggests.

The root cause is a missing guard, not a bad number. MEASURED_ON_NODE_MAJOR
pins the runtime input to the measurement and fails loudly when it moves.
The suite is the other input — every region in the table is credited by some
spec executing it — and nothing pins it, so the record rots silently each time a
spec is added.

Note this is not a coverage gap: src/** is at 100% unit lines and regions, and
100% e2e source lines. The gate itself is sound; only its written record is
wrong.

Recommendation

  • Refresh the table with the re-measured figures above, and restate the
    headroom paragraph in terms of the aggregate (where the ratchet risk
    actually lives) rather than the per-file floor.
  • Add a MEASURED_AT_SPEC_COUNT constant asserted against
    readdirSync('tests/e2e'), so the record cannot drift from the suite
    silently again. A spec count is the cheapest hermetic proxy: no browser,
    no build, no coverage run, and it moves exactly when the suite does.
    Drive both failure arms from fixtures, matching the file's existing idiom
    that "a guard nobody has seen go red is indistinguishable from one that
    always passes".

Both are tests/-only; no workflow file and no floor value changes.

Priority

  • Impact: medium — nothing is red today, but the record exists specifically to
    prevent a bad ratchet of a required check, and it currently argues for one.
  • Effort: low — the measurements are in the table above.

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: deb268d

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

Dominant language
JavaScript
Stars
0
Forks
2
Avg merge
10h 13m
Merged PRs (30d)
597

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cncf/endusers

All issues in cncf/endusers

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.