[quality] The READ_ONLY_VALIDATORS list is the only PR-time data gate, and nothing keeps it in sync with scripts/validate-*.mjs

Open Beginner friendly
#365 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript
Domain
ci-cd, testing-qa

Research direction

Read tests/validators-smoke.test.mjs and inspect the scripts/validate-*.mjs files; start by running the existing validator smoke test and the full test suite. Add tests/validator-smoke-coverage.test.mjs so the listed validators and read-only validator files match in both directions, then verify the clean suite and the new-validator failure case.

Written by the indexing model from the issue text.

Description

agent/quality hive/hosted-available-lke648397-260827-5n31 quality testing

Finding

tests/validators-smoke.test.mjs holds a hand-maintained array of validator
filenames:

const READ_ONLY_VALIDATORS = [
  'validate-metrics.mjs',
  'validate-awards.mjs',
  'validate-architectures.mjs',
  'validate-architecture-assets.mjs',
  'validate-button-contrast.mjs',
];

Nothing keeps that list in sync with scripts/validate-*.mjs, and the list is
far more load-bearing than a unit-test fixture normally is.

It is the only thing that gates data integrity on a pull request.

.github/workflows/ci.yml is the sole pull_request-triggered workflow. Its
steps are npm ci, npm run test:unit, npm run build:production — it never
invokes npm run validate:*. The four data validators run directly only in
deploy-gh-pages.yml (trigger: push to main) and import-architectures.yml
(trigger: schedule). Both are post-merge.

So the validators reach the PR gate exclusively via
validators-smoke.test.mjs, which executes each listed script against the real
repository data.

Verified at 00b44df (node v26.8.1)

Mutating data/awards.json to give the first entry an http:// announcement
URL and running node --test:

✖ validate-awards.mjs passes against current repo data
  stderr: '\n1 error(s) in awards:\n  [error] 2026/sncf: announcementUrl must be https\n'

The unit suite catches it — but only because validate-awards.mjs happens to
appear in that array. A validator added to scripts/ and forgotten in the
array runs on no pull request. Its first failure lands post-merge on the
push-to-main deploy, where it blocks deployment of an already-merged change,
and the author who introduced the break is no longer in the loop.

Why existing gates miss it
  • The array is a literal list of strings. Adding scripts/validate-foo.mjs
    leaves all 55 existing tests green.
  • docusaurus build does not model the test runner's inputs.
  • PR #278 (tests/workflow-scripts.test.mjs, open) asserts that npm run
    targets named in workflows and package.json resolve to defined scripts and
    existing files. It does not read validators-smoke.test.mjs and does not
    model which validators actually execute on a pull request.
  • PR #301 (tests/ci-supply-chain.test.mjs, open) covers action pinning and
    token permissions, not step composition.
  • PR #225 (tests/tools/coverage-report.mjs, open) changes how coverage is
    reported for fixture-sandbox runs; it does not touch the smoke list.
  • No open PR touches tests/validators-smoke.test.mjs.

Recommendation

  • Add tests/validator-smoke-coverage.test.mjs, which parses the
    READ_ONLY_VALIDATORS literal out of tests/validators-smoke.test.mjs
    and asserts, in both directions, that it matches the read-only
    scripts/validate-*.mjs files on disk.

"Read-only" is detected from the script source rather than hard-coded, so the
guard does not go stale when a validator gains a write mode.
validate-architecture-assets.mjs has a --fix path and is therefore not
required to be listed, though it is listed today and that stays valid.

Verified: green on a clean checkout (4/4 pass, full suite 59/59), and dropping
a read-only scripts/validate-foo.mjs into the tree fails it with
actual: [ 'validate-foo.mjs' ]. Already prettier --check clean.

This is test-only — no production file changes.

Priority

  • Impact: medium
  • Effort: low

Filed by quality agent (hold-gated mode)

— hive: agent=quality backend=copilot model=claude-opus-5

Dominant language
JavaScript
Stars
0
Forks
2
Avg merge
2d 22h
Merged PRs (30d)
12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cncf/endusers

All issues in cncf/endusers

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.