Add AGENTS.md guidance: flag legacy project removal when adding app-next lanes
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- playwright
- Domain
- documentation, testing
Research direction
Update the E2E Testing review guidance in AGENTS.md, using docs/nfs-e2e-triage.md and the outcome of PR #3342 as context. Add the app-next guidance and migration-status checks described in the issue. Done when the guidance tells reviewers when to flag legacy project removal and how to handle unclear migration status.
Written by the indexing model from the issue text.
Description
What happened
PR #3342 added an orchestrator-app-next Playwright project alongside the existing orchestrator project. The review agent ran 12 successful reviews (first review, through final review) and approved the PR multiple times, but never questioned whether the legacy project was still needed. Human reviewer jrichter1 posted a single inline comment asking "can we just get rid of the old frontend system now?" The author responded within 40 minutes by dropping the legacy project (commit 7ef2936), which halved the test matrix from 64 to 32 tests and eliminated the need for flock-based Loki install serialization (commit b2f2113).
What could go better
The review agent lacked guidance to question whether an existing Playwright project is superseded when a new -app-next project is added for the same workspace. The repo already tracks NFS migration status in docs/nfs-e2e-triage.md (21 legacy lanes, 28 NFS lanes at merge time), but this information is not referenced during review. The insight required knowing the orchestrator plugin had fully migrated to NFS — product knowledge the agent did not have. Confidence is high that this is a repeatable gap: the same pattern will recur as other workspaces complete their NFS migration, and without guidance the review agent will continue approving PRs that add app-next projects without flagging legacy removal. Six other workspaces currently carry both legacy and app-next projects, so the opportunity to catch redundant projects is concrete.
Proposed change
Add a section to the E2E Testing review guidance in AGENTS.md (or a linked review-guidance document) with content like:
### Adding app-next Playwright projects
When a PR adds an `-app-next` Playwright project to a workspace that already has a legacy (non-app-next) project:
1. Check `docs/nfs-e2e-triage.md` to determine whether the workspace's NFS migration is complete.
2. If the workspace is listed as NFS-complete (all tests passing on the app-next lane), flag that the legacy project should be removed rather than kept alongside the new one. Keeping both doubles CI cost and introduces cross-project race conditions (shared Loki installs, Knative CRDs) that require serialization workarounds.
3. If the migration status is unclear, ask the author whether the legacy lane is still needed.
This gives the review agent a concrete heuristic to apply without requiring deep product knowledge.
Validation criteria
On the next PR that adds an -app-next Playwright project to a workspace already tracked as NFS-complete in docs/nfs-e2e-triage.md, the review agent should flag whether the legacy project can be removed. Verify against the next 2–3 such PRs within 90 days. Success means the agent raises the question before a human does (or the PR already removes the legacy project).
Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3342
- Dominant language
- TypeScript
- Stars
- 9
- Forks
- 72
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 133
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from redhat-developer/rhdh-plugin-export-overlays
-
documentation non-workspace-changes ready-for-triage ready-to-code
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3815 · 3 comments ·
-
documentation non-workspace-changes ready-for-triage ready-to-code
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
redhat-developer/rhdh-plugin-export-overlays#3810 · 3 comments ·
-
Add AGENTS.md review guidance: Prettier/ESLint/TypeScript violations in e2e-tests are CI-blocking Opendocumentation ready-for-triage ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3792 · 3 comments ·
-
e2e-failure ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3789 · 1 comment ·
-
e2e-failure ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3788 · 1 comment ·
All issues in redhat-developer/rhdh-plugin-export-overlays
Similar issues
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100