garrytan/gbrain

Budget-cap denial retries in a tight loop with no backoff and no abort (20,276 retries in 23h)

Offen

#3.669 geöffnet am 31.07.2026

 (2 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)TypeScript (3.057 Forks)github user discovery
fix-neededhelp wantedp1verified-real

Repository-Metriken

Stars
 (21.368 Sterne)
PR-Merge-Metriken
 (Durchschn. Merge 18T 10h) (162 gemergte PRs in 30 T)

Beschreibung

Severity: high (pathological CPU/log burn; a cap makes things worse than no cap)

What happened. cycle.conversation_facts_backfill hit its per-run cost cap and then retried the same denied reservation continuously for ~23 hours instead of aborting the phase.

Verified counts — all 20,276 reserve_denied events sit in a single weekly ledger:

~/.gbrain/audit/budget-2026-W30.jsonl   6.2 MB   reserve_denied = 20276
(every other budget-*.jsonl and dream-budget-*.jsonl: 0)

Window: 2026-07-24T06:34:58Z2026-07-25T05:26:20Z~880 denials/hour, roughly one every 4 seconds, for 23 hours straight.

A representative event:

{
  "ts": "2026-07-24T06:34:58.942Z",
  "event": "reserve_denied",
  "label": "conversation_facts_backfill:brain-wide",
  "kind": "chat",
  "model": "anthropic:claude-sonnet-4-6",
  "sub_label": "gateway.chat",
  "projected_cost_usd": 0.06276,
  "cumulative_cost_usd": 0.44122,
  "max_cost_usd": 0.5
}

The arithmetic is stable and unwinnable: 0.44122 + 0.06276 = 0.50398 > 0.50. The projected cost never shrinks, so the reservation can never succeed — yet it is retried 20k times. The 6.2 MB ledger is itself a symptom.

Related design note. A cap set below one work unit's cost is strictly worse than no cap: the phase burns CPU and log volume forever while accomplishing zero work and spending $0.

Correction (2026-07-26, same session): an earlier draft of this file claimed cycle.propose_takes was in that same state — "halted at page 0, $0 spent." That was wrong and is retracted. The ledger shows propose_takes runs ~21–26×/day (autopilot is KeepAlive with ThrottleInterval 60), the cap is per run, not per day, and each run spends up to ~$0.048 before denial — ~$1.00–1.25/day, not $0. Its 292 submit_denied events are normal bounded cap behaviour, NOT the busy-loop described above. The 20,276-event busy-loop is conversation_facts_backfill only. Do not conflate the two phases.

Asks.

  1. On reserve_denied, abort the phase (or exponential-backoff) rather than retry immediately.
  2. Detect the unwinnable case at phase start: if max_cost_usd < projected_unit_cost, refuse to start with a clear message naming both numbers, instead of spinning.
  3. Rate-limit or dedupe identical denial events in the audit ledger.

Contributor Guide