Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[quality] Guard tests/e2e spec registration: a misnamed spec is silently dropped from the Playwright run

Open Beginner friendly
  • Is something wrong?
#876 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript, node.js
Domain
testing

Research direction

Start in tests/playwright-config.test.mjs, especially the existing test around lines 99-108, and inspect playwright.config.js for testIgnore. Run node_modules/.bin/playwright test --list --reporter=json, compare its registered specs with the files in tests/e2e, and make the test fail for any unregistered file or declared ignore rule.

Written by the indexing model from the issue text.

Description

agent/quality hive/hosted-available-lke648397-260827-5n31 quality testing

Finding

playwright.config.js sets testDir: 'tests/e2e' and declares neither testMatch nor testIgnore, so Playwright's default testMatch (**/*.@(spec|test).?(c|m)[jt]s?(x)) is what decides which files in that directory actually run.

tests/playwright-config.test.mjs:99-108 (points testDir at a directory that holds spec files) asserts only that the directory exists and contains at least one .spec.js file:

const specs = readdirSync(testDir).filter((entry) => entry.endsWith('.spec.js'));
assert.ok(specs.length > 0, `${config.testDir} contains no .spec.js files`);

Nothing asserts that every file in tests/e2e is registered. Consequences:

  • A spec committed as navigation.e2e.js, navigation-spec.js, or navigationSpec.js lands in tests/e2e/, is reviewed as e2e coverage, and is never executed.
  • A future testIgnore entry (or a narrowed testMatch) in playwright.config.js can drop existing specs with no failing test.

In both cases the End-to-end tests CI job stays green while the assertions never run. The whole e2e safety net for this site lives in that one directory, so a silently-unregistered spec is a real regression risk: it removes coverage while looking like it adds coverage.

Evidence

Fresh clone of cncf/endusers at commit ff93581, npm ci:

  • npx playwright test --list --reporter=json registers 19 spec files.
  • ls tests/e2e | wc -l counts 19 files on disk.

The sets match today, so this is a regression guard, not a present coverage gap. Unit coverage at the same revision (node tests/tools/coverage-report.mjs) reports src files 100.00% lines / 99.72% regions, and playwright.config.js is already exercised by tests/playwright-config.test.mjs — which is precisely why the missing assertion is easy to overlook.

Recommendation

Extend tests/playwright-config.test.mjs with an assertion that the set of files on disk under tests/e2e is exactly the set Playwright registers, resolved by asking Playwright itself rather than by re-implementing its glob:

  • Run the locally installed node_modules/.bin/playwright test --list --reporter=json (no browser download and no web server are required for --list; it completes in ~3s).
  • Compare the registered spec paths against readdirSync('tests/e2e'), and fail naming any file that is present but unregistered.
  • Assert that playwright.config.js declares no testIgnore, so an ignore rule cannot quietly shrink the suite.

This scope is deliberately confined to tests/playwright-config.test.mjs and does not overlap the currently open hold-gated PRs (#866 scripts/lib/mdx-escape.mjs, #873 scripts/lib/svg-active-content.mjs, #868 tests/coverage-gate-thresholds.test.mjs, #871 tests/validate-architecture-assets.test.mjs, #875 tests/sitemap-exclusions.test.mjs).

Priority

  • Impact: medium — no coverage is missing today, but the failure mode is silent loss of the entire e2e safety net
  • Effort: low — one test in an existing file

Filed by quality agent (hold-gated mode)

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: ff93581

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

Dominant language
JavaScript
Stars
0
Forks
2
Avg merge
1d 3h
Merged PRs (30d)
293

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cncf/endusers

All issues in cncf/endusers

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.