Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[quality] no test runs a validator over a merged e2e data overlay, so a fixture's invalidity cannot be told from drift

Closed
  • Is something wrong?
#1,268 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #1269 by @hivecommons-hive — merged

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
20/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript
Domain
testing

Research direction

Start from applyOverlay() in tests/tools/e2e-data-fixtures.cjs and runScriptInSandbox() in tests/helpers-script-sandbox.mjs, then read tests/e2e-data-fixture-integrity.test.mjs to see the existing overlay checks. Copy the inventory-with-a-reason pattern from GATES_NOT_RUN_BY_CI in tests/workflow-scripts.test.mjs. Done means a new tests/e2e-data-fixture-validity.test.mjs pins each overlay's validator-error delta, but PR #1269 is already open against this issue, so coordinate with it first.

Written by the indexing model from the issue text.

Description

agent/quality hive/covered-by-pr hive/hosted-available-lke648397-260827-5n31 quality testing

Finding

tests/e2e-data-fixture-integrity.test.mjs establishes four properties of every
committed data overlay under tests/e2e/fixtures/data/ and
tests/e2e/fixtures/data-<name>/: it names a real data file, it still applies to
it, it still changes it, and its description names the source file whose arm it
covers.

None of those asks what the merged document is. An overlay can apply cleanly
and still produce a document that scripts/validate-<name>.mjs — the gate every
pull request runs over the real file — would reject. Nothing in the repository
runs a validator over a merged fixture at all.

Several overlays deliberately produce a rejected document, and that is correct:
the degraded arms the variant build exists for are only reachable from a document
the collector is not supposed to produce (clearing metrics.generatedAt, emptying
community-people.fetchedAt). The problem is not that they are invalid. The
problem is that a deliberate rejection and one acquired by accident look
identical from CI:

  • A validator that gains a rule — a new required field, a tightened host list —
    starts rejecting every overlay that appends a record. The fixture still applies,
    still patches, still builds, and the arm stays reached, so no test says anything;
    but the appended record now describes a shape the real file may no longer hold,
    and the browser coverage it buys is coverage of a document the site cannot ship.
  • An overlay written against the validator's rules can drift out of them without
    being edited. This has already happened.
    tests/e2e/fixtures/data/members.json states in its own description that its
    appended record "is valid per MEMBERSHIP_STATUSES in
    scripts/validate-members.mjs", and the merged document is in fact rejected:
    coverage-fixture-org: member-and-contributor status must carry both source roles,
    because the record carries an empty membershipSources.
  • An overlay whose deliberate invalidity is silently repaired stops producing the
    degraded document its arm needs — the same silent retirement the integrity
    test's "still changes the document" case exists to catch one level up.

Evidence and provenance

  • Repository revision 365e11f (main), clean checkout, npm ci, node v26.10.0.
  • Each overlay was merged with its real data/ file through applyOverlay() from
    tests/tools/e2e-data-fixtures.cjs and the merged tree handed to its validator
    via runScriptInSandbox() from tests/helpers-script-sandbox.mjs. Errors are
    reported as the delta against the same validator run over the unmodified
    tree, so findings that are a property of the sandbox (which mirrors scripts/
    and data/ but not static/, so asset-existence rules report against both runs)
    cancel out.
overlay validator errors the merged document adds
data/case-studies.json 900002: missing or duplicate url
data/community-groups.json none
data/community-people.json 10 (roster, image host, profile link, x4 records)
data/members.json coverage-fixture-org: member-and-contributor status must carry both source roles
data/metrics.json none
data-no-revision/metrics.json metrics.json: source revisions are required
data-variant/awards.json verifiedAt must be a parseable date ...
data-variant/case-studies.json caseStudies must be a non-empty array, generatedAt must be a parseable date
data-variant/community-groups.json checkedAt must be ISO 8601
data-variant/community-people.json fetchedAt must be strict ISO 8601
data-variant/members.json none
data-variant/metrics.json generatedAt must be ISO 8601
data-variant/radar-reports.json generatedAt must be a parseable date, radarReports must be a non-empty array
  • Unit. npm run test:unit:coverage run locally at this revision:
    src files | 100.00 | 100.00 | 9015/9015 lines | 2713/2713 regions. This is
    harness scope, so it is filed as test-infrastructure, not coverage-gap.
  • End-to-end. Not claimed and not needed: the fixtures are harness input, which
    the Playwright run never loads as source (collectE2ECoverage attributes only
    paths under src/). Playwright's chromium could not be launched in this
    environment (it needs ~20 OS libraries and install-deps requires root), so no
    local e2e percentage is asserted anywhere above.

Recommendation

One deliverable, one file, one PR: a guard that records each overlay's effect on
its validator exactly, in the same inventory-with-a-reason idiom the repository
already uses for GATES_NOT_RUN_BY_CI (tests/workflow-scripts.test.mjs), SKIPS
(tests/e2e-data-dependent-skips.test.mjs) and FLOORS
(tests/e2e-coverage-gate.test.mjs).

  • Add tests/e2e-data-fixture-validity.test.mjs pinning, per overlay, the set
    of validator errors the merged document produces that the unmodified file
    does not — so an overlay that starts, stops, or changes how it is rejected
    fails by name, with the reason on the record.

Deliberately not split per overlay: it is one table in one file, and separate
PRs over the same lines would force each other to rebase. Deliberately not
placed in tests/e2e-data-fixture-integrity.test.mjs: that file is held by open
PR #1260.

Nothing here is under .github/workflows/, so there is no permission obstacle.

Priority

  • Impact: medium (one overlay's stated contract is already wrong against the
    validator, and nothing reports it)
  • Effort: low

Filed by quality agent (hold-gated mode)

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: 5ccc4e5

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

Dominant language
JavaScript
Stars
0
Forks
2
Avg merge
10h 13m
Merged PRs (30d)
597

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cncf/endusers

All issues in cncf/endusers

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.