Eval runner + shipped Claude Code adapter: fails for every run as root, waits 3 s per run on stdin, and absolute skill_dir leaks synthetic skills

オープン
  • 何か問題がありますか？
#158 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
backend, testing

調査の方向性

Start with skills/bmad-eval-runner/scripts/run_evals.py, run_triggers.py, assets/adapter-claude-code.json, and platform-adapter.md. Reproduce the root execution and piped-stdin cases, then trace how skill_dir is staged and cleaned up. Done means root runs fail clearly or use a supported permission mode, subprocesses do not wait for stdin, and invalid paths are rejected without leaving synthetic skills behind.

索引モデルが issue の本文から書いたものです。

説明

Describe the bug

I tested skills/bmad-eval-runner/ with the shipped assets/adapter-claude-code.json against real Claude Code 2.1.322 and found three problems.

1. Every run fails when the runner executes as root. The adapter invokes claude -p {prompt} … --dangerously-skip-permissions. Claude Code refuses that flag under root with "--dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons" and exits 1. Root is the default user in Docker and many CI containers, which are exactly the "clean, isolated" environments people pick for evals. Because of #122 the failure is invisible. A real trigger run as root scored the explicit query "Use the yaml validator skill to validate this YAML: a: [1, 2" (should_trigger: true) at 0.0, scored the negative as a pass, set no status, and exited 0.

To confirm detection itself is fine: the same query without the flag produced tool_use Skill {"skill": "yaml-validator-trig-abc12345", ...}, which detect_load() matches.

2. Each run waits 3 s for stdin. subprocess.run(...) in run_evals.py and run_triggers.py doesn't set stdin, so the child inherits the runner's stdin. When that is a non-TTY pipe, as it is under an agent or CI, claude -p waits and logs "Warning: no stdin data received in 3s, proceeding without it." That adds 3 s to every case and every trigger run, about 15 minutes for 100 queries × 3 runs.

3. An absolute skill_dir leaks synthetic skills into a real skills folder. platform-adapter.md says skill_dir is relative to the cwd, but nothing enforces it. With an absolute value, stage_dir / skill_dir is that absolute path. write_synthetic_skill() writes <name>-trig-<uuid> skills there, and the cleanup rmtree(stage / skill_dir.split("/")[0]) removes stage/"" instead. One trigger run with 2 queries × 3 runs left 6 synthetic skills in the target folder. Pointed at ~/.claude/skills, they would load in every future session. A skill_dir beginning with .. makes the same rmtree delete sibling query stages mid-run.

Steps to reproduce

# (1) as root, with the shipped adapter
python3 skills/bmad-eval-runner/scripts/run_triggers.py --skill-path skill/yaml-validator \
  --queries q.json --output-dir out --adapter skills/bmad-eval-runner/assets/adapter-claude-code.json --runs-per-query 2
# should_trigger query: trigger_rate 0.0; negative: pass; no status; exit 0
# the same command line by hand prints: --dangerously-skip-permissions cannot be used with root/sudo privileges

# (2) any run from a non-TTY parent: claude logs "no stdin data received in 3s" per run

# (3) absolute skill_dir
echo '{"invocation":["/bin/echo","{}"],"skill_dir":"/tmp/global-skills"}' > abs.json
python3 skills/bmad-eval-runner/scripts/run_triggers.py --skill-path skill/my-skill --queries q.json --output-dir out --adapter abs.json --runs-per-query 3
ls /tmp/global-skills     # my-skill-trig-xxxxxxxx  x6, never removed

Expected behavior

  1. Detect root and switch to a permission mode Claude Code accepts there, e.g. --permission-mode with an explicit allow-list, or document a non-root requirement. Fail loudly (see #122 / #135) instead of scoring failures.
  2. Pass stdin=subprocess.DEVNULL in both runners.
  3. Reject absolute or .. values for skill_dir (see #136), and clean up the exact directory the synthetic skill was written to, not the first path segment.

Environment

  • Model(s) used: n/a (deterministic script) / claude-sonnet-5-5 via Claude Code 2.1.322 for the live runs
  • Agentic IDE used: n/a (scripts run directly) / Claude Code CLI 2.1.322 as the eval runtime
  • BMad version: bmad-builder main @ d4a921c (v2.2.2 + #118)
  • Project language: Python 3.13, Linux
主要言語
Python
スター
220
フォーク
54
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

bmad-code-org/bmad-builder のほかの issue

bmad-code-org/bmad-builder の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。