Add AGENTS.md review guidance: verify data pipeline reachability for scripts gating on structured values
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 74/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- documentation
Research direction
Start with AGENTS.md, then read scripts/check_app_next_lanes.py and docs/nfs-readiness-report.json to understand the existing review guidance and current readiness data. Add the proposed section covering reachable upstream values, inert blocking verdicts, and NFS status verification; done means the guidance clearly directs future reviews to validate gate behavior against current data.
Written by the indexing model from the issue text.
Description
What happened
PR #3362 added a CI gate (check_app_next_lanes.py) that classified workspaces by NFS readiness status. The review agent approved all three commits (run 1, run 2, run 3), praising the three-verdict design and claiming the status sets were exhaustive. The author then closed the PR without merging, explaining that mixed and legacy-only statuses are structurally unreachable because rhdh-cli populates backstage.features from MF entry points. The gate's blocked verdict depended on values the readiness report never produces. With corrected data from PR #3361, the gate would flag zero of the ten workspaces with active -app-next lanes. The review agent could not catch this because it lacked domain knowledge about the NFS readiness data pipeline's actual output space.
What could go better
The review agent reviewed code quality dimensions thoroughly (logic correctness, testing, security, style) but did not question whether the input data supports the code's assumptions. When a script classifies data by specific values and gates on the classification, the review should verify that those values are actually reachable in the current data pipeline — otherwise the gate produces no signal (as happened here) or, worse, produces false positives (as would have happened with PR #3343, which the gate would have incorrectly blocked).
This is a hard problem for static code review: the code's logic was internally correct given its assumptions, and the assumptions were plausible based on reading the shell script code. The flaw was only visible when understanding that rhdh-cli populates backstage.features from MF entry points rather than from independent classification. This is domain knowledge that the review agent did not have.
Confidence: High that AGENTS.md guidance would help. The NFS readiness pipeline is actively evolving (20+ related PRs, PR #3361 adds finer-grained classification), and future gate PRs are expected once the report carries nfsFeaturesExposed. The review agent already follows AGENTS.md guidance closely in this repo.
Uncertainty: The guidance may not fully prevent the miss if the upstream data behavior is not documented in this repo. However, instructing the reviewer to ask "can this gate's blocking verdict actually fire given current data?" would prompt the right investigation.
Proposed change
Add a review guidance section to AGENTS.md under a heading like "Reviewing scripts that gate on structured pipeline data". The guidance should instruct the review agent:
- When a script in
scripts/consumes structured data from an internal pipeline (NFS readiness report, catalog index, smoke test results) and classifies items by specific values (statuses, categories, types), verify that the upstream data source actually produces all values the script's blocking logic depends on. - Forward-compatibility handlers for currently unreachable values are acceptable, but blocking verdicts (non-zero exit, CI gate failures) must not depend solely on values that the current data pipeline cannot produce.
- For NFS readiness data specifically: the readiness report's output space is determined by
rhdh-cli, which populatesbackstage.featuresfrom module-federation entry points. The set of reachable statuses may be narrower than what the report schema allows. When reviewing NFS gate scripts, check the actual status distribution in the latest readiness report (docs/nfs-readiness-report.jsonor the wiki report) before accepting that blocking logic is reachable. - When a gate would produce zero blocking verdicts against all current data (i.e., the gate is inert), flag this as a correctness concern — an inert gate provides false confidence that the check is active.
This guidance is specific to the NFS readiness pipeline area, which is the most active area in the repo (20+ PRs under RHIDP-15286) and where future gate attempts are explicitly planned.
Validation criteria
The next review of a script that gates on NFS readiness statuses (or similar structured pipeline data) should include verification that the gate's blocking conditions can actually be triggered by current data. Specifically: if a revised version of PR #3362 is submitted after the readiness report carries nfsFeaturesExposed (per RHIDP-16463), the review should verify the gate's verdict logic against the updated data schema and confirm the gate produces non-trivial results against the current readiness dataset. The review should not approve a gate that is inert against all current data without flagging this.
Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3362
- Dominant language
- TypeScript
- Stars
- 9
- Forks
- 72
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 133
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from redhat-developer/rhdh-plugin-export-overlays
-
documentation non-workspace-changes ready-for-triage ready-to-code
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3815 · 3 comments ·
-
documentation non-workspace-changes ready-for-triage ready-to-code
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
redhat-developer/rhdh-plugin-export-overlays#3810 · 3 comments ·
-
Add AGENTS.md review guidance: Prettier/ESLint/TypeScript violations in e2e-tests are CI-blocking Opendocumentation ready-for-triage ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3792 · 3 comments ·
-
e2e-failure ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3789 · 1 comment ·
-
e2e-failure ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3788 · 1 comment ·
All issues in redhat-developer/rhdh-plugin-export-overlays
Similar issues
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100