issueAsk rail demotes a reviewer APPROVE for an ask that is present in the diff

Open
#1,760 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
go, kubernetes

Research direction

Start by reading enforceReviewerIssueAsk in pkg/foreman/agent/executor_native.go and the related guard in internal/foreman/controller/workload_iteration.go. Trace how reviewer verdicts and issue ask verification are combined, then inspect relevant tests or add a regression case for an explicit APPROVE with an unverified ask. Done means the rail no longer turns that approval into a rejection without a basis.

Written by the indexing model from the issue text.

Description

bug

Bug Description

enforceReviewerIssueAsk (pkg/foreman/agent/executor_native.go) rewrote a reviewer's APPROVE to NO-GO because it could not verify the stated issue ask — while that ask was implemented in the diff under review, at four separate call sites.

The rail's own hazard is already documented at internal/foreman/controller/workload_iteration.go:175:

That is a statement about verification confidence, not about the change... re-running the coder cannot make an unverifiable issueAsk verify, so the loop has no terminating condition.

The controller guards against that by refusing to open a fix iteration (workload_iteration.go:220). But nothing prevents the demotion itself, so a correct, approved change is turned into a rejection, no PR is opened, and any external retry loop spends its whole budget re-running work that was already right.

Steps to Reproduce

  1. File an issue whose body contains a numbered "Suggested shape" list, so one list item reads as a discrete, checkable ask.
  2. Let a coder implement it and return GO.
  3. Let the reviewer review it and reach APPROVE.

Observed on misospace/dispatch#940, an issue whose body had a four-point suggestion list.

Expected Behavior

A rail that cannot verify an ask should not convert an explicit reviewer APPROVE into a rejection. Either it downgrades to a warning and lets the reviewer's own verdict stand, or the demotion is gated on the reviewer having actually had the means to verify.

Actual Behavior

Audit record for the review task:

"verdict":  "NO-GO",
"reviewer": { "outcome": "APPROVE" },
"issueAsk": { "verified": false,
              "claimed": "Record `headSha` at enqueue so a comparison is possible at all." },
"turnCount": 18,
"elapsedSec": 184.8

The claimed string is item 1 of the issue's "Suggested shape" list. The coder's branch implements exactly that:

02bab1e fix(pr-fix-queue): refuse FIXED on no-progress and reopen stale tombstones (#940)
 8 files changed, 450 insertions(+), 20 deletions(-)

with headSha: prHeadSha written at enqueue in four call sites. So the rail marked unverified a change sitting in the diff it was reviewing.

Downstream: no PR (the reviewer opens one only on GO), three identical coder attempts at turnCount 158 each, and the issue parked for a human. The 450 lines are still on the branch, unmerged.

Possibly relevant

The reviewer ran 18 turns in 185s. I could not confirm what it did — transcriptRef points at foreman-transcript-wl-misospace-dispatch-940-review-940-0, which does not exist in the namespace, so the transcript for this review is unavailable.

A separate review in the same cluster stated in its own output:

The native tool-calling loop issued no tool calls; review was performed against the provided corpus only.

If a reviewer reaches its verdict without issuing tool calls, it has no means to verify an ask against the diff, and verified: false would be the expected outcome for every ask rather than a signal about this change. That is a hypothesis, not something I can demonstrate for this run.

Environment

  • LLMKube operator and coder image 0.9.24
  • Agent reviewer, mode: InProcess, maxTurns: 80
  • Model: gemma-4-12b-it-qat via a LiteLLM proxy
  • contextStrategy: session, contextWindowTokens: 170000

Note

Filed from operating a Foreman deployment; happy to gather more from this cluster if a specific field or code path would help. The missing review transcript may itself be worth a look — an audit record whose transcriptRef resolves to nothing removes the only evidence of how a verdict was reached.

Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.