Add AGENTS.md review guidance: verify data pipeline reachability for scripts gating on structured values

Open Beginner friendly
#3,387 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Domain
documentation

Research direction

Start with AGENTS.md, then read scripts/check_app_next_lanes.py and docs/nfs-readiness-report.json to understand the existing review guidance and current readiness data. Add the proposed section covering reachable upstream values, inert blocking verdicts, and NFS status verification; done means the guidance clearly directs future reviews to validate gate behavior against current data.

Written by the indexing model from the issue text.

Description

documentation ready-for-triage ready-to-code

What happened

PR #3362 added a CI gate (check_app_next_lanes.py) that classified workspaces by NFS readiness status. The review agent approved all three commits (run 1, run 2, run 3), praising the three-verdict design and claiming the status sets were exhaustive. The author then closed the PR without merging, explaining that mixed and legacy-only statuses are structurally unreachable because rhdh-cli populates backstage.features from MF entry points. The gate's blocked verdict depended on values the readiness report never produces. With corrected data from PR #3361, the gate would flag zero of the ten workspaces with active -app-next lanes. The review agent could not catch this because it lacked domain knowledge about the NFS readiness data pipeline's actual output space.

What could go better

The review agent reviewed code quality dimensions thoroughly (logic correctness, testing, security, style) but did not question whether the input data supports the code's assumptions. When a script classifies data by specific values and gates on the classification, the review should verify that those values are actually reachable in the current data pipeline — otherwise the gate produces no signal (as happened here) or, worse, produces false positives (as would have happened with PR #3343, which the gate would have incorrectly blocked).

This is a hard problem for static code review: the code's logic was internally correct given its assumptions, and the assumptions were plausible based on reading the shell script code. The flaw was only visible when understanding that rhdh-cli populates backstage.features from MF entry points rather than from independent classification. This is domain knowledge that the review agent did not have.

Confidence: High that AGENTS.md guidance would help. The NFS readiness pipeline is actively evolving (20+ related PRs, PR #3361 adds finer-grained classification), and future gate PRs are expected once the report carries nfsFeaturesExposed. The review agent already follows AGENTS.md guidance closely in this repo.

Uncertainty: The guidance may not fully prevent the miss if the upstream data behavior is not documented in this repo. However, instructing the reviewer to ask "can this gate's blocking verdict actually fire given current data?" would prompt the right investigation.

Proposed change

Add a review guidance section to AGENTS.md under a heading like "Reviewing scripts that gate on structured pipeline data". The guidance should instruct the review agent:

  1. When a script in scripts/ consumes structured data from an internal pipeline (NFS readiness report, catalog index, smoke test results) and classifies items by specific values (statuses, categories, types), verify that the upstream data source actually produces all values the script's blocking logic depends on.
  2. Forward-compatibility handlers for currently unreachable values are acceptable, but blocking verdicts (non-zero exit, CI gate failures) must not depend solely on values that the current data pipeline cannot produce.
  3. For NFS readiness data specifically: the readiness report's output space is determined by rhdh-cli, which populates backstage.features from module-federation entry points. The set of reachable statuses may be narrower than what the report schema allows. When reviewing NFS gate scripts, check the actual status distribution in the latest readiness report (docs/nfs-readiness-report.json or the wiki report) before accepting that blocking logic is reachable.
  4. When a gate would produce zero blocking verdicts against all current data (i.e., the gate is inert), flag this as a correctness concern — an inert gate provides false confidence that the check is active.

This guidance is specific to the NFS readiness pipeline area, which is the most active area in the repo (20+ PRs under RHIDP-15286) and where future gate attempts are explicitly planned.

Validation criteria

The next review of a script that gates on NFS readiness statuses (or similar structured pipeline data) should include verification that the gate's blocking conditions can actually be triggered by current data. Specifically: if a revised version of PR #3362 is submitted after the readiness report carries nfsFeaturesExposed (per RHIDP-16463), the review should verify the gate's verdict logic against the updated data schema and confirm the gate produces non-trivial results against the current readiness dataset. The review should not approve a gate that is inert against all current data without flagging this.


Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3362

Dominant language
TypeScript
Stars
9
Forks
72
Avg merge
3d 9h
Merged PRs (30d)
133

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from redhat-developer/rhdh-plugin-export-overlays

All issues in redhat-developer/rhdh-plugin-export-overlays

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.