Add AGENTS.md guidance: flag legacy project removal when adding app-next lanes

Open Beginner friendly
#3,482 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
playwright

Research direction

Update the E2E Testing review guidance in AGENTS.md, using docs/nfs-e2e-triage.md and the outcome of PR #3342 as context. Add the app-next guidance and migration-status checks described in the issue. Done when the guidance tells reviewers when to flag legacy project removal and how to handle unclear migration status.

Written by the indexing model from the issue text.

Description

documentation ready-for-triage ready-to-code

What happened

PR #3342 added an orchestrator-app-next Playwright project alongside the existing orchestrator project. The review agent ran 12 successful reviews (first review, through final review) and approved the PR multiple times, but never questioned whether the legacy project was still needed. Human reviewer jrichter1 posted a single inline comment asking "can we just get rid of the old frontend system now?" The author responded within 40 minutes by dropping the legacy project (commit 7ef2936), which halved the test matrix from 64 to 32 tests and eliminated the need for flock-based Loki install serialization (commit b2f2113).

What could go better

The review agent lacked guidance to question whether an existing Playwright project is superseded when a new -app-next project is added for the same workspace. The repo already tracks NFS migration status in docs/nfs-e2e-triage.md (21 legacy lanes, 28 NFS lanes at merge time), but this information is not referenced during review. The insight required knowing the orchestrator plugin had fully migrated to NFS — product knowledge the agent did not have. Confidence is high that this is a repeatable gap: the same pattern will recur as other workspaces complete their NFS migration, and without guidance the review agent will continue approving PRs that add app-next projects without flagging legacy removal. Six other workspaces currently carry both legacy and app-next projects, so the opportunity to catch redundant projects is concrete.

Proposed change

Add a section to the E2E Testing review guidance in AGENTS.md (or a linked review-guidance document) with content like:

### Adding app-next Playwright projects

When a PR adds an `-app-next` Playwright project to a workspace that already has a legacy (non-app-next) project:

1. Check `docs/nfs-e2e-triage.md` to determine whether the workspace's NFS migration is complete.
2. If the workspace is listed as NFS-complete (all tests passing on the app-next lane), flag that the legacy project should be removed rather than kept alongside the new one. Keeping both doubles CI cost and introduces cross-project race conditions (shared Loki installs, Knative CRDs) that require serialization workarounds.
3. If the migration status is unclear, ask the author whether the legacy lane is still needed.

This gives the review agent a concrete heuristic to apply without requiring deep product knowledge.

Validation criteria

On the next PR that adds an -app-next Playwright project to a workspace already tracked as NFS-complete in docs/nfs-e2e-triage.md, the review agent should flag whether the legacy project can be removed. Verify against the next 2–3 such PRs within 90 days. Success means the agent raises the question before a human does (or the PR already removes the legacy project).


Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3342

Dominant language
TypeScript
Stars
9
Forks
72
Avg merge
3d 9h
Merged PRs (30d)
133

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from redhat-developer/rhdh-plugin-export-overlays

All issues in redhat-developer/rhdh-plugin-export-overlays

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.