Add AGENTS.md guidance: fix agent must verify CI status before declaring success

Open Beginner friendly
#3,684 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
1-3 hours
Newbie friendliness
88/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active

Research direction

Open AGENTS.md and locate the “E2E Nightly Fix Conventions” section. Add the requested “Verifying fix effectiveness” subsection using the four stated CI-verification rules, then check that the guidance matches the validation criteria for the next three relevant fix-agent invocations.

Written by the indexing model from the issue text.

Description

documentation non-workspace-changes ready-for-triage ready-to-code

What happened

On PR #3545, a human invoked /fs-fix on Sep 7 pointing to a triage analysis comment that said the PR's dynamic model discovery approach already covered both VLLM outage and model rename scenarios. The fix agent ran for 6 minutes ($0.92), read the triage analysis, agreed with its conclusion, made zero code changes, and declared success referencing the original commit 3d40357.

However, the E2E test modified by the PR was deterministically failing on CI. After /publish on Sep 7, the E2E run showed 14 passed, 1 failed — the exact test the fix was supposed to address. The same failure reproduced on Sep 8. The ci-diagnose agent later confirmed it was a PR regression caused by incorrect API response assumptions, wrong model name extraction logic, and a locator context change.

The fix agent trusted the triage bot's analysis over observable CI evidence. The triage analysis evaluated the PR's conceptual approach (dynamic discovery) but not its implementation correctness. The fix agent did not independently check CI results or investigate the actual test failure.

What could go better

The fix agent should treat a deterministic CI failure as the strongest signal available. When invoked to fix a CI failure, finding a triage comment that says "no changes needed" should not override the fact that CI is still failing on the PR. The fix agent should have:

  1. Checked whether CI was passing on the current PR commit before accepting any analysis at face value
  2. If CI was failing, investigated the actual failure output rather than relying solely on the triage analysis
  3. Attempted a code fix addressing the specific failure, or explicitly flagged that it could not resolve the issue

Confidence: High. The fix agent's behavior is clearly documented in the PR timeline — it declared success with the original commit SHA, and CI failed deterministically on the same test afterward. The gap between "triage says it's fine" and "CI says it's broken" is unambiguous.

Proposed change

Add a new subsection to the "E2E Nightly Fix Conventions" section of AGENTS.md titled "Verifying fix effectiveness" with this guidance:

  • When the fix agent is invoked to address a CI failure and determines no code changes are needed, it MUST check whether CI is currently passing for the PR before declaring success.
  • If CI results show a deterministic failure in the same test or workspace the fix targets, the fix agent should investigate the actual failure (test output, error messages, assertion mismatches) rather than trusting triage analysis alone.
  • The fix agent should not declare success with the original commit SHA when the test it was asked to fix is still failing on CI. Instead, it should either attempt a code fix or explicitly report that it cannot resolve the failure and explain why.
  • When a triage analysis says "no changes needed" but CI disagrees, CI evidence takes precedence — triage evaluates the approach, CI evaluates the implementation.

This guidance belongs in AGENTS.md (not .fullsend/ config) because the code/fix agent already reads AGENTS.md at runtime and the E2E Nightly Fix Conventions section is the established location for fix agent behavioral guidance.

Validation criteria

On the next 3 fix agent invocations for E2E failures in this repo where the fix agent considers making no changes: (1) the fix agent explicitly checks CI status on the PR before declaring success, (2) if CI is failing on the target test, the fix agent either pushes a code fix or reports inability to fix with specific reasoning about the failure mechanism, and (3) the fix agent does not declare success referencing an unchanged commit while CI is still failing.


Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3545

Dominant language
TypeScript
Stars
9
Forks
72
Avg merge
3d 9h
Merged PRs (30d)
133

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from redhat-developer/rhdh-plugin-export-overlays

All issues in redhat-developer/rhdh-plugin-export-overlays

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.