Seed a Claude Code usage-limit/quota env-fault pattern once a real line is captured

Cerrado
  • ¿Algo no está bien?
#610 4 comentarios 2 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
devtools, testing

Línea de trabajo

Empieza por src/bmad_loop/data/profiles/claude.toml y tests/test_env_fault_patterns.py, especialmente test_unseeded_profiles_stay_inert y el corpus CLAUDE_REAL. Revisa EnvFaultMixin y las pruebas de tests/test_generic_tmux.py antes de decidir si una línea capturada de Claude Code es suficiente o si se necesita una fuente para escanear transcripciones. Se considera terminado cuando una línea real citada se clasifica sin falsos positivos de salida saludable y todos los gates del corpus mencionados pasan.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:adapters bug help wanted P4

Follow-up to #323 and #507, whose PR replaced claude.toml's env_fault_patterns. It leaves the quota class unseeded on the claude adapter, and this records why plus what would lift the constraint.

What is missing

src/bmad_loop/data/profiles/claude.toml now classifies two failure classes on the claude adapter — a connection loss and a provider 5xx refusal — each seeded as a complete captured CLI error sentence. It classifies no subscription usage-limit refusal. So when an Anthropic plan's usage limit stops a session, the run still charges a dev attempt, exhausts max_dev_attempts and defers a story whose spec was fine. That is the same failure #323 reported on another provider, and it is still unshipped on this adapter.

Why it was not done in the #323/#507 PR

There is no captured Claude Code usage-limit line to seed from.

Searched on 2026-08-15:

  • this repo's tracked tree;
  • issues #194, #323 and #507, bodies and comments;
  • 2681 local Claude Code JSONL transcripts under ~/.claude/projects (2684 across all of ~/.claude).

usage limit reached and limit will reset do hit in the transcripts — 33 and 31 files respectively — but every hit is a citation, not an emission: issue #323's own AI_APICallError: Usage limit reached for 5 hour… line, which is opencode's provider text rather than Claude Code's; this repo's tests/test_env_fault_patterns.py corpus; the withdrawn draft pattern; and agent sessions discussing exactly those. None is a line Claude Code printed.

The standing bar lives in tests/test_env_fault_patterns.py, in test_unseeded_profiles_stay_inert's docstring: seed from a captured log line with the run it came from cited, never from a plausible-looking string. An unverified pattern is not a neutral bet — one that fires on a healthy session pauses the entire run, which is worse than the fault it was meant to catch.

Why the vocabulary cannot simply be added

The claude adapter's scanned log is a tmux pane capture (EnvFaultMixin.ENV_FAULT_LOG_SUFFIX), so it carries the model's own output. 429, rate limit, quota and usage limit are exactly what a story that implements rate limiting prints all day — in test names, fixtures, banner copy, acceptance criteria and diff hunks.

That is measured, not assumed. Compiling the narrowest plausible quota anchor, API Error: 429, against the corpus already in tests/test_env_fault_patterns.py false-positives on 3 lines sitting in ANCHOR_REACHING_BAIT today:

// TODO: surface "API Error: 429" to the user with a Retry-After hint
expect(banner).toBe("API Error: 429 rate_limit_error")
- [ ] AC-2: map API Error: 429 to a friendly rate limit message

A slightly looser form (the framing token plus up to 80 characters plus quota vocabulary) takes it to 4. Every one of those is ordinary healthy-session output, and every one would pause a working run.

This is the same trap #507 documented from the other side: widening a cause list onto a loose anchor makes the false-positive half worse, because quota vocabulary is far more common in ordinary story text than ECONNREFUSED is.

An option that would remove the constraint — offered as an option, not a commitment

Point the claude profile's env-fault scan at Claude Code's own JSONL transcript instead of the pane capture, and anchor on the structural key "isApiErrorMessage":true rather than on error vocabulary at all.

Why that anchor differs in kind: inside the transcript a model's own words live within a JSON string, so a citation of the key is escaped — \"isApiErrorMessage\" — while the structural key is not. A citation cannot physically reproduce the anchor. That is the property #507 asked for, and it is not available on a pane capture at all.

Measured across the same 2681 transcripts:

form occurrences files
unescaped structural key "isApiErrorMessage":true 11 7
JSON-escaped citation \"isApiErrorMessage\" 46 4

Zero file overlap. And the 11 structural occurrences carry only 4 distinct texts, all genuine CLI errors:

  • API Error: 500 Internal server error…
  • API Error: 529 Overloaded…
  • API Error: Connection closed mid-response…
  • a model-selection error (There's an issue with the selected model…)

— despite those same files containing plenty of prose citing provider errors.

The cost, stated honestly, because it is not small:

  • a new per-profile field naming which source the env-fault scan reads;
  • a scan-source seam in EnvFaultMixin: _env_fault_evidence() takes only a task_id today and derives the path from ENV_FAULT_LOG_SUFFIX, whereas the transcript path arrives later, on the SessionResult;
  • porting the ~15 test_classify_env_fault_* tests in tests/test_generic_tmux.py, which write their fixture to whatever path the code resolves;
  • it is claude-specific — no other pane-capture profile has an equivalent structural transcript.

Worth weighing too: the transcript is written by the CLI, so repointing trades one soundness argument for a different one, and that argument has to be re-derived rather than inherited.

What to do when a line IS captured

  1. Add the captured line to CLAUDE_REAL in tests/test_env_fault_patterns.py, citing the run it came from.
  2. Extend claude.toml's env_fault_patterns with a pattern reproducing that complete sentence, keeping a single-character class so the pattern does not match its own source line.
  3. Re-run the corpus — test_seeded_profiles_never_classify_healthy_model_output, test_seeded_profiles_survive_bait_that_reaches_their_anchor, test_shipped_patterns_do_not_match_their_own_profile_line and test_pane_capture_patterns_do_not_match_this_repo are the gate.

Context: #323 (the quota-miss half) and #507 (the false-positive half).

Lenguaje dominante
Python
Estrellas
174
Forks
72
Merge medio
19 h 25 min
PR fusionados (30 d)
84

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de bmad-code-org/bmad-loop

Todos los issues de bmad-code-org/bmad-loop

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.