harness: level-triggered forever watch can stampede duplicate Agent Runs
#953 opened on Jul 22, 2026
Repository metrics
- Stars
- (489 stars)
- PR merge metrics
- (Avg merge 4h 21m) (286 merged PRs in 30d)
Description
Summary
A managed vibe watch --forever can create an unbounded burst of duplicate Agent Runs when its waiter is level-triggered and keeps returning exit code 0 after a condition becomes true.
This is easy to hit when an Agent needs "retry until ready, then resume once": retry exit codes are only retried in forever mode, but forever also repeats after success. The two lifecycle requirements are currently coupled.
Observed incident
On 2026-07-22, a watch polled a local provider-readiness predicate:
- while not ready, it returned
75and used a 30-second retry delay; - once ready, it returned
0and remained true; - the watch targeted one existing Agent Session.
After the predicate became true, the watch created 274 real follow-up runs in 18.139 seconds (about 15 runs/second):
- 174 were dequeued and marked succeeded, mostly as fast no-op turns;
- 100 remained queued and had to be canceled manually;
- the watch itself had to be removed to stop further growth.
The numbers above exclude the synthetic runtime:<definition_id> run row. No credentials, prompts, or user data are included here.
Current behavior
The successful-cycle path:
- enqueues a follow-up for every exit code
0; - disables only
oncewatches; - immediately
continues forforeverwatches, with no delay or edge detection.
See core/watches.py lines 674-692.
Each event then creates a fresh run with no coalescing by definition/session. See core/watches.py lines 793-816 and core/scheduled_tasks.py lines 947-980.
retry_delay_seconds applies to configured retry exit codes, but not to successful cycles.
Minimal reproduction shape
- Create a
foreverwatch whose command returns75until a local condition is true. - Make the condition true so the command returns
0on every subsequent invocation. - Observe
vibe runs list --definition-id <watch-id>and followpagination.next_commandwhile more pages exist.
The run count grows continuously until the watch is paused or removed.
Expected behavior
Avibe should safely support "retry until the first success, enqueue one follow-up, then stop" without requiring every Agent to hand-roll a blocking/edge-triggered waiter.
A single configuration mistake should also not create unbounded Agent Runs.
Suggested acceptance criteria
- Add an explicit
until-successlifecycle, or allowoncewatches to retry configured retry exit codes but stop after the first0. - Prevent more than one queued/running follow-up for the same watch definition and target Session, or coalesce repeated events while one is in flight.
- Add a success-cycle interval, rate limit, or circuit breaker for
foreverwatches so a persistent0cannot spin. - Preserve legitimate edge-triggered
foreverwatches that block until each new event. - Add a regression test with a waiter that returns persistent
0and assert bounded follow-up creation. - Surface a warning or terminal error when a watch exceeds a short-window trigger threshold.
Root-cause classification
The incident was triggered by an Agent choosing the wrong waiter shape, but the blast radius is a Harness safety/design issue: forever conflates retry-until-ready with repeat-after-success, and the queue has no deduplication, backpressure, or fuse for one watch definition.