Seed a Claude Code usage-limit/quota env-fault pattern once a real line is captured

Đã đóng
  • Có gì không ổn?
#610 4 bình luận 2 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
devtools, testing

Hướng nghiên cứu

Bắt đầu với src/bmad_loop/data/profiles/claude.toml và tests/test_env_fault_patterns.py, đặc biệt là test_unseeded_profiles_stay_inert và corpus CLAUDE_REAL. Xem xét EnvFaultMixin và các test trong tests/test_generic_tmux.py trước khi quyết định liệu một dòng Claude Code được thu thập có đủ hay cần một nguồn để quét transcript. Được xem là hoàn tất khi một dòng thực tế được trích dẫn được phân loại mà không có false positive đối với đầu ra lành mạnh và tất cả các corpus gate được nêu đều vượt qua.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area:adapters bug help wanted P4

Follow-up to #323 and #507, whose PR replaced claude.toml's env_fault_patterns. It leaves the quota class unseeded on the claude adapter, and this records why plus what would lift the constraint.

What is missing

src/bmad_loop/data/profiles/claude.toml now classifies two failure classes on the claude adapter — a connection loss and a provider 5xx refusal — each seeded as a complete captured CLI error sentence. It classifies no subscription usage-limit refusal. So when an Anthropic plan's usage limit stops a session, the run still charges a dev attempt, exhausts max_dev_attempts and defers a story whose spec was fine. That is the same failure #323 reported on another provider, and it is still unshipped on this adapter.

Why it was not done in the #323/#507 PR

There is no captured Claude Code usage-limit line to seed from.

Searched on 2026-08-15:

  • this repo's tracked tree;
  • issues #194, #323 and #507, bodies and comments;
  • 2681 local Claude Code JSONL transcripts under ~/.claude/projects (2684 across all of ~/.claude).

usage limit reached and limit will reset do hit in the transcripts — 33 and 31 files respectively — but every hit is a citation, not an emission: issue #323's own AI_APICallError: Usage limit reached for 5 hour… line, which is opencode's provider text rather than Claude Code's; this repo's tests/test_env_fault_patterns.py corpus; the withdrawn draft pattern; and agent sessions discussing exactly those. None is a line Claude Code printed.

The standing bar lives in tests/test_env_fault_patterns.py, in test_unseeded_profiles_stay_inert's docstring: seed from a captured log line with the run it came from cited, never from a plausible-looking string. An unverified pattern is not a neutral bet — one that fires on a healthy session pauses the entire run, which is worse than the fault it was meant to catch.

Why the vocabulary cannot simply be added

The claude adapter's scanned log is a tmux pane capture (EnvFaultMixin.ENV_FAULT_LOG_SUFFIX), so it carries the model's own output. 429, rate limit, quota and usage limit are exactly what a story that implements rate limiting prints all day — in test names, fixtures, banner copy, acceptance criteria and diff hunks.

That is measured, not assumed. Compiling the narrowest plausible quota anchor, API Error: 429, against the corpus already in tests/test_env_fault_patterns.py false-positives on 3 lines sitting in ANCHOR_REACHING_BAIT today:

// TODO: surface "API Error: 429" to the user with a Retry-After hint
expect(banner).toBe("API Error: 429 rate_limit_error")
- [ ] AC-2: map API Error: 429 to a friendly rate limit message

A slightly looser form (the framing token plus up to 80 characters plus quota vocabulary) takes it to 4. Every one of those is ordinary healthy-session output, and every one would pause a working run.

This is the same trap #507 documented from the other side: widening a cause list onto a loose anchor makes the false-positive half worse, because quota vocabulary is far more common in ordinary story text than ECONNREFUSED is.

An option that would remove the constraint — offered as an option, not a commitment

Point the claude profile's env-fault scan at Claude Code's own JSONL transcript instead of the pane capture, and anchor on the structural key "isApiErrorMessage":true rather than on error vocabulary at all.

Why that anchor differs in kind: inside the transcript a model's own words live within a JSON string, so a citation of the key is escaped — \"isApiErrorMessage\" — while the structural key is not. A citation cannot physically reproduce the anchor. That is the property #507 asked for, and it is not available on a pane capture at all.

Measured across the same 2681 transcripts:

form occurrences files
unescaped structural key "isApiErrorMessage":true 11 7
JSON-escaped citation \"isApiErrorMessage\" 46 4

Zero file overlap. And the 11 structural occurrences carry only 4 distinct texts, all genuine CLI errors:

  • API Error: 500 Internal server error…
  • API Error: 529 Overloaded…
  • API Error: Connection closed mid-response…
  • a model-selection error (There's an issue with the selected model…)

— despite those same files containing plenty of prose citing provider errors.

The cost, stated honestly, because it is not small:

  • a new per-profile field naming which source the env-fault scan reads;
  • a scan-source seam in EnvFaultMixin: _env_fault_evidence() takes only a task_id today and derives the path from ENV_FAULT_LOG_SUFFIX, whereas the transcript path arrives later, on the SessionResult;
  • porting the ~15 test_classify_env_fault_* tests in tests/test_generic_tmux.py, which write their fixture to whatever path the code resolves;
  • it is claude-specific — no other pane-capture profile has an equivalent structural transcript.

Worth weighing too: the transcript is written by the CLI, so repointing trades one soundness argument for a different one, and that argument has to be re-derived rather than inherited.

What to do when a line IS captured

  1. Add the captured line to CLAUDE_REAL in tests/test_env_fault_patterns.py, citing the run it came from.
  2. Extend claude.toml's env_fault_patterns with a pattern reproducing that complete sentence, keeping a single-character class so the pattern does not match its own source line.
  3. Re-run the corpus — test_seeded_profiles_never_classify_healthy_model_output, test_seeded_profiles_survive_bait_that_reaches_their_anchor, test_shipped_patterns_do_not_match_their_own_profile_line and test_pane_capture_patterns_do_not_match_this_repo are the gate.

Context: #323 (the quota-miss half) and #507 (the false-positive half).

Ngôn ngữ chính
Python
Star
174
Fork
72
Merge trung bình
19 giờ 25 phút
Pull request đã merge (30 ngày)
84

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của bmad-code-org/bmad-loop

Tất cả issue của bmad-code-org/bmad-loop

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.