Agent work is lost on abnormal stage termination — watcher pushes but never checkpoints uncommitted work

Aperta
  • Qualcosa non va?
#166 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@hhpatel14 ci sta già lavorando.

Dal 20/8/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug
Summary

When a stage terminates abnormally (OOMKill, timeout, pod eviction, cancel), all work the agent did in that stage is silently lost because it was never committed. The filesystem watcher pushes on every change but only pushes committed history; the agent commits just once at end-of-stage, so anything before that final commit is discarded on a hard kill — and nothing records that the stage did any work.

Observed

In an AgentWorkflowRun e2e (Java EE → Quarkus), the verify stage applied 12 compile-fix edits (ShippingService, ShoppingCart, CartEndpoint, OrderEndpoint, ProductEndpoint, application.properties, …), then was OOMKilled during a blackbox mvn quarkus:dev.

Result: 0 commits from verify on the branch, and no record of the edits anywhere in status. The three prior stages (questionnaire/plan/execute), which each reached their end-of-stage commit, persisted fine.

$ kubectl logs <run>-verify -c agent | grep -c "tool: edit ·"   # 12
$ kubectl logs <run>-verify -c agent | grep -c "git commit"     # 0
Mechanism
  • harness/internal/watcher/watcher.go — doPush() only calls pushFn; it never commits.
  • harness/internal/git/git.go:216 — Push skips when HEAD == baseSHA (no commits produced; skipping push).
  • The agent is the sole committer, once, at stage end (via the ## Working Guidelines prompt). SIGKILL (OOM) can't be trapped, so a deferred "final commit" on exit wouldn't run either.

Net effect: the watcher gives a false sense of safety — it reports watcher push: success on every change while actually persisting nothing, because there is nothing committed yet.

Asks
  1. Checkpoint uncommitted work. Have the harness periodically (and/or on the watcher's debounced events) git add -A && git commit a WIP checkpoint before pushing, so persisted state tracks the agent's actual progress and survives a hard kill.
  2. Report work-in-progress on failure. Surface what the stage produced (files changed / last checkpoint) in the AgentRun status instead of a bare failure, so a killed stage isn't a black hole.
Design tension

This touches the ADR 0007 / ADR 0010 "agent commits locally, harness pushes" contract — harness-authored checkpoint commits are a deliberate deviation (e.g. harness: checkpoint commits that the agent's real end-of-stage commit later supersedes/squashes). Flagging for discussion; may warrant an ADR.

Related secondary gaps (same incident, lower priority)

The above is the primary issue. The OOM that triggered it also exposed two smaller gaps, noted here for context:

  • Agent pods run BestEffort — the Agent/AgentRun CRDs expose no resources field and the controller sets no requests/limits, so pods have no memory guarantees and are first to be OOM-killed under node pressure.
  • Failure reason isn't surfaced — internal/controller/agentrun_controller.go:644 reports the generic Sandbox finished with reason: PodFailed and never reads the pod's containerStatuses[].state.terminated.reason/exitCode, so OOMKilled (exit 137) is only discoverable by manually inspecting the pod.
Lingua principale
Go
Stelle
5
Fork
13
Merge medio
3g 3h
PR unite (30g)
45

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di konveyor/agentic-controller

Tutte le issue di konveyor/agentic-controller

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.