Eval runner + shipped Claude Code adapter: fails for every run as root, waits 3 s per run on stdin, and absolute skill_dir leaks synthetic skills

Aperta
  • Qualcosa non va?
#158 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, testing

Direzione di ricerca

Start with skills/bmad-eval-runner/scripts/run_evals.py, run_triggers.py, assets/adapter-claude-code.json, and platform-adapter.md. Reproduce the root execution and piped-stdin cases, then trace how skill_dir is staged and cleaned up. Done means root runs fail clearly or use a supported permission mode, subprocesses do not wait for stdin, and invalid paths are rejected without leaving synthetic skills behind.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

I tested skills/bmad-eval-runner/ with the shipped assets/adapter-claude-code.json against real Claude Code 2.1.322 and found three problems.

1. Every run fails when the runner executes as root. The adapter invokes claude -p {prompt} … --dangerously-skip-permissions. Claude Code refuses that flag under root with "--dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons" and exits 1. Root is the default user in Docker and many CI containers, which are exactly the "clean, isolated" environments people pick for evals. Because of #122 the failure is invisible. A real trigger run as root scored the explicit query "Use the yaml validator skill to validate this YAML: a: [1, 2" (should_trigger: true) at 0.0, scored the negative as a pass, set no status, and exited 0.

To confirm detection itself is fine: the same query without the flag produced tool_use Skill {"skill": "yaml-validator-trig-abc12345", ...}, which detect_load() matches.

2. Each run waits 3 s for stdin. subprocess.run(...) in run_evals.py and run_triggers.py doesn't set stdin, so the child inherits the runner's stdin. When that is a non-TTY pipe, as it is under an agent or CI, claude -p waits and logs "Warning: no stdin data received in 3s, proceeding without it." That adds 3 s to every case and every trigger run, about 15 minutes for 100 queries × 3 runs.

3. An absolute skill_dir leaks synthetic skills into a real skills folder. platform-adapter.md says skill_dir is relative to the cwd, but nothing enforces it. With an absolute value, stage_dir / skill_dir is that absolute path. write_synthetic_skill() writes <name>-trig-<uuid> skills there, and the cleanup rmtree(stage / skill_dir.split("/")[0]) removes stage/"" instead. One trigger run with 2 queries × 3 runs left 6 synthetic skills in the target folder. Pointed at ~/.claude/skills, they would load in every future session. A skill_dir beginning with .. makes the same rmtree delete sibling query stages mid-run.

Steps to reproduce

# (1) as root, with the shipped adapter
python3 skills/bmad-eval-runner/scripts/run_triggers.py --skill-path skill/yaml-validator \
  --queries q.json --output-dir out --adapter skills/bmad-eval-runner/assets/adapter-claude-code.json --runs-per-query 2
# should_trigger query: trigger_rate 0.0; negative: pass; no status; exit 0
# the same command line by hand prints: --dangerously-skip-permissions cannot be used with root/sudo privileges

# (2) any run from a non-TTY parent: claude logs "no stdin data received in 3s" per run

# (3) absolute skill_dir
echo '{"invocation":["/bin/echo","{}"],"skill_dir":"/tmp/global-skills"}' > abs.json
python3 skills/bmad-eval-runner/scripts/run_triggers.py --skill-path skill/my-skill --queries q.json --output-dir out --adapter abs.json --runs-per-query 3
ls /tmp/global-skills     # my-skill-trig-xxxxxxxx  x6, never removed

Expected behavior

  1. Detect root and switch to a permission mode Claude Code accepts there, e.g. --permission-mode with an explicit allow-list, or document a non-root requirement. Fail loudly (see #122 / #135) instead of scoring failures.
  2. Pass stdin=subprocess.DEVNULL in both runners.
  3. Reject absolute or .. values for skill_dir (see #136), and clean up the exact directory the synthetic skill was written to, not the first path segment.

Environment

  • Model(s) used: n/a (deterministic script) / claude-sonnet-5-5 via Claude Code 2.1.322 for the live runs
  • Agentic IDE used: n/a (scripts run directly) / Claude Code CLI 2.1.322 as the eval runtime
  • BMad version: bmad-builder main @ d4a921c (v2.2.2 + #118)
  • Project language: Python 3.13, Linux
Lingua principale
Python
Stelle
220
Fork
54
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di bmad-code-org/bmad-builder

Tutte le issue di bmad-code-org/bmad-builder

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.