Add AGENTS.md guidance: fix agent must verify CI status before declaring success
Nobody has claimed this yet.
Assessment
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 88/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- ci-cd, documentation
Research direction
Open AGENTS.md and locate the “E2E Nightly Fix Conventions” section. Add the requested “Verifying fix effectiveness” subsection using the four stated CI-verification rules, then check that the guidance matches the validation criteria for the next three relevant fix-agent invocations.
Written by the indexing model from the issue text.
Description
What happened
On PR #3545, a human invoked /fs-fix on Sep 7 pointing to a triage analysis comment that said the PR's dynamic model discovery approach already covered both VLLM outage and model rename scenarios. The fix agent ran for 6 minutes ($0.92), read the triage analysis, agreed with its conclusion, made zero code changes, and declared success referencing the original commit 3d40357.
However, the E2E test modified by the PR was deterministically failing on CI. After /publish on Sep 7, the E2E run showed 14 passed, 1 failed — the exact test the fix was supposed to address. The same failure reproduced on Sep 8. The ci-diagnose agent later confirmed it was a PR regression caused by incorrect API response assumptions, wrong model name extraction logic, and a locator context change.
The fix agent trusted the triage bot's analysis over observable CI evidence. The triage analysis evaluated the PR's conceptual approach (dynamic discovery) but not its implementation correctness. The fix agent did not independently check CI results or investigate the actual test failure.
What could go better
The fix agent should treat a deterministic CI failure as the strongest signal available. When invoked to fix a CI failure, finding a triage comment that says "no changes needed" should not override the fact that CI is still failing on the PR. The fix agent should have:
- Checked whether CI was passing on the current PR commit before accepting any analysis at face value
- If CI was failing, investigated the actual failure output rather than relying solely on the triage analysis
- Attempted a code fix addressing the specific failure, or explicitly flagged that it could not resolve the issue
Confidence: High. The fix agent's behavior is clearly documented in the PR timeline — it declared success with the original commit SHA, and CI failed deterministically on the same test afterward. The gap between "triage says it's fine" and "CI says it's broken" is unambiguous.
Proposed change
Add a new subsection to the "E2E Nightly Fix Conventions" section of AGENTS.md titled "Verifying fix effectiveness" with this guidance:
- When the fix agent is invoked to address a CI failure and determines no code changes are needed, it MUST check whether CI is currently passing for the PR before declaring success.
- If CI results show a deterministic failure in the same test or workspace the fix targets, the fix agent should investigate the actual failure (test output, error messages, assertion mismatches) rather than trusting triage analysis alone.
- The fix agent should not declare success with the original commit SHA when the test it was asked to fix is still failing on CI. Instead, it should either attempt a code fix or explicitly report that it cannot resolve the failure and explain why.
- When a triage analysis says "no changes needed" but CI disagrees, CI evidence takes precedence — triage evaluates the approach, CI evaluates the implementation.
This guidance belongs in AGENTS.md (not .fullsend/ config) because the code/fix agent already reads AGENTS.md at runtime and the E2E Nightly Fix Conventions section is the established location for fix agent behavioral guidance.
Validation criteria
On the next 3 fix agent invocations for E2E failures in this repo where the fix agent considers making no changes: (1) the fix agent explicitly checks CI status on the PR before declaring success, (2) if CI is failing on the target test, the fix agent either pushes a code fix or reports inability to fix with specific reasoning about the failure mechanism, and (3) the fix agent does not declare success referencing an unchanged commit while CI is still failing.
Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3545
- Dominant language
- TypeScript
- Stars
- 9
- Forks
- 72
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 133
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from redhat-developer/rhdh-plugin-export-overlays
-
documentation non-workspace-changes ready-for-triage ready-to-code
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3815 · 3 comments ·
-
documentation non-workspace-changes ready-for-triage ready-to-code
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
redhat-developer/rhdh-plugin-export-overlays#3810 · 3 comments ·
-
Add AGENTS.md review guidance: Prettier/ESLint/TypeScript violations in e2e-tests are CI-blocking Opendocumentation ready-for-triage ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3792 · 3 comments ·
-
e2e-failure ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3789 · 1 comment ·
-
e2e-failure ready-to-code
Difficulty 1/5 Under an hour Newbie friendliness 88/100
redhat-developer/rhdh-plugin-export-overlays#3788 · 1 comment ·
All issues in redhat-developer/rhdh-plugin-export-overlays
Similar issues
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100