Seed a Claude Code usage-limit/quota env-fault pattern once a real line is captured

Chiusa
  • Qualcosa non va?
#610 4 commenti 2 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
devtools, testing

Direzione di ricerca

Inizia da src/bmad_loop/data/profiles/claude.toml e tests/test_env_fault_patterns.py, in particolare test_unseeded_profiles_stay_inert e il corpus CLAUDE_REAL. Esamina EnvFaultMixin e i test in tests/test_generic_tmux.py prima di decidere se sia sufficiente una riga catturata di Claude Code o se serva una sorgente per analizzare le trascrizioni. Il lavoro è completato quando una riga reale citata viene classificata senza falsi positivi nell’output sano e tutti i gate del corpus indicati vengono superati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:adapters bug help wanted P4

Follow-up to #323 and #507, whose PR replaced claude.toml's env_fault_patterns. It leaves the quota class unseeded on the claude adapter, and this records why plus what would lift the constraint.

What is missing

src/bmad_loop/data/profiles/claude.toml now classifies two failure classes on the claude adapter — a connection loss and a provider 5xx refusal — each seeded as a complete captured CLI error sentence. It classifies no subscription usage-limit refusal. So when an Anthropic plan's usage limit stops a session, the run still charges a dev attempt, exhausts max_dev_attempts and defers a story whose spec was fine. That is the same failure #323 reported on another provider, and it is still unshipped on this adapter.

Why it was not done in the #323/#507 PR

There is no captured Claude Code usage-limit line to seed from.

Searched on 2026-08-15:

  • this repo's tracked tree;
  • issues #194, #323 and #507, bodies and comments;
  • 2681 local Claude Code JSONL transcripts under ~/.claude/projects (2684 across all of ~/.claude).

usage limit reached and limit will reset do hit in the transcripts — 33 and 31 files respectively — but every hit is a citation, not an emission: issue #323's own AI_APICallError: Usage limit reached for 5 hour… line, which is opencode's provider text rather than Claude Code's; this repo's tests/test_env_fault_patterns.py corpus; the withdrawn draft pattern; and agent sessions discussing exactly those. None is a line Claude Code printed.

The standing bar lives in tests/test_env_fault_patterns.py, in test_unseeded_profiles_stay_inert's docstring: seed from a captured log line with the run it came from cited, never from a plausible-looking string. An unverified pattern is not a neutral bet — one that fires on a healthy session pauses the entire run, which is worse than the fault it was meant to catch.

Why the vocabulary cannot simply be added

The claude adapter's scanned log is a tmux pane capture (EnvFaultMixin.ENV_FAULT_LOG_SUFFIX), so it carries the model's own output. 429, rate limit, quota and usage limit are exactly what a story that implements rate limiting prints all day — in test names, fixtures, banner copy, acceptance criteria and diff hunks.

That is measured, not assumed. Compiling the narrowest plausible quota anchor, API Error: 429, against the corpus already in tests/test_env_fault_patterns.py false-positives on 3 lines sitting in ANCHOR_REACHING_BAIT today:

// TODO: surface "API Error: 429" to the user with a Retry-After hint
expect(banner).toBe("API Error: 429 rate_limit_error")
- [ ] AC-2: map API Error: 429 to a friendly rate limit message

A slightly looser form (the framing token plus up to 80 characters plus quota vocabulary) takes it to 4. Every one of those is ordinary healthy-session output, and every one would pause a working run.

This is the same trap #507 documented from the other side: widening a cause list onto a loose anchor makes the false-positive half worse, because quota vocabulary is far more common in ordinary story text than ECONNREFUSED is.

An option that would remove the constraint — offered as an option, not a commitment

Point the claude profile's env-fault scan at Claude Code's own JSONL transcript instead of the pane capture, and anchor on the structural key "isApiErrorMessage":true rather than on error vocabulary at all.

Why that anchor differs in kind: inside the transcript a model's own words live within a JSON string, so a citation of the key is escaped — \"isApiErrorMessage\" — while the structural key is not. A citation cannot physically reproduce the anchor. That is the property #507 asked for, and it is not available on a pane capture at all.

Measured across the same 2681 transcripts:

form occurrences files
unescaped structural key "isApiErrorMessage":true 11 7
JSON-escaped citation \"isApiErrorMessage\" 46 4

Zero file overlap. And the 11 structural occurrences carry only 4 distinct texts, all genuine CLI errors:

  • API Error: 500 Internal server error…
  • API Error: 529 Overloaded…
  • API Error: Connection closed mid-response…
  • a model-selection error (There's an issue with the selected model…)

— despite those same files containing plenty of prose citing provider errors.

The cost, stated honestly, because it is not small:

  • a new per-profile field naming which source the env-fault scan reads;
  • a scan-source seam in EnvFaultMixin: _env_fault_evidence() takes only a task_id today and derives the path from ENV_FAULT_LOG_SUFFIX, whereas the transcript path arrives later, on the SessionResult;
  • porting the ~15 test_classify_env_fault_* tests in tests/test_generic_tmux.py, which write their fixture to whatever path the code resolves;
  • it is claude-specific — no other pane-capture profile has an equivalent structural transcript.

Worth weighing too: the transcript is written by the CLI, so repointing trades one soundness argument for a different one, and that argument has to be re-derived rather than inherited.

What to do when a line IS captured

  1. Add the captured line to CLAUDE_REAL in tests/test_env_fault_patterns.py, citing the run it came from.
  2. Extend claude.toml's env_fault_patterns with a pattern reproducing that complete sentence, keeping a single-character class so the pattern does not match its own source line.
  3. Re-run the corpus — test_seeded_profiles_never_classify_healthy_model_output, test_seeded_profiles_survive_bait_that_reaches_their_anchor, test_shipped_patterns_do_not_match_their_own_profile_line and test_pane_capture_patterns_do_not_match_this_repo are the gate.

Context: #323 (the quota-miss half) and #507 (the false-positive half).

Lingua principale
Python
Stelle
174
Fork
72
Merge medio
19h 25m
PR unite (30g)
84

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di bmad-code-org/bmad-loop

Tutte le issue di bmad-code-org/bmad-loop

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.