Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

evals: run-validity calls runs VALID when every Bash call failed to start the sandbox (long TMPDIR)

Open Beginner friendly
  • Is something wrong?
#9,611 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@kyle-sexton is already working on this.

Since Oct 25, 2026.

  • #10096 by @kyle-sexton — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
shell
Domain
devtools, testing

Research direction

Start with plugins/evals/skills/plugin-eval/scripts/run-validity.py and inspect how it reads kept traces and evaluates tool results; compare that with the reported sandbox initialization failure. Then read plugins/evals/skills/plugin-eval/reference/sandboxes.md for the TMPDIR guidance. Done when validity flags or warns on these failed Bash calls and the reference explains the path constraint by pointing to CLI documentation.

Written by the indexing model from the issue text.

Description

priority: medium status: needs-decision work-class: scoped

What happened

claude plugin eval (Claude Code 2.1.310, WSL2) with TMPDIR set to a directory on disk at a 50-character path (<home>/<dir>/<dir>/eval-tmp) ran 150 session-flow runs (--scaffold --keep-temp --runs 3 --allow-tools Bash Write Edit).

  • 129 of the 150 kept traces show every Bash call failing with Sandbox is required but failed to initialize: Failed to create bridge sockets after 5 attempts. Restart to retry. Running it outside the sandbox (dangerouslyDisableSandbox) is not allowed either. The runs finished normally (error: null), and their scores measured a session with no shell.
  • plugins/evals/skills/plugin-eval/scripts/run-validity.py reported verdict: VALID for 20 of the 25 cases. Its run-error check reads the run's error field and the trace's final state, not tool-result errors.
  • A one-case probe isolates the cause: a Bash-only case fails the same way at that TMPDIR. The same case prints its output at a 24-character TMPDIR. Likely the sandbox's socket paths under $TMPDIR/claude-eval-<id>/tmp/... exceed the Unix socket path limit (UNVERIFIED; the exact limit was not measured).

Expected

  • run-validity.py reports a run as invalid, or at least warns, when its trace holds a sandbox start failure on a tool call.
  • The plugin-eval skill says that a long TMPDIR breaks the sandbox, with the limit stated by pointer to wherever the CLI documents it.

Suggested fix location

  • plugins/evals/skills/plugin-eval/scripts/run-validity.py: a sandbox check that scans kept traces for Sandbox is required but failed to initialize.
  • plugins/evals/skills/plugin-eval/reference/sandboxes.md: the TMPDIR length note.

Related

  • #9514: the same gate also misses 0-turn runs caused by a sandbox write failure.

Found while running the session-flow handoff-efficiency baselines (branch feat/session-flow-handoff-efficiency).

Dominant language
Shell
Stars
22
Forks
2
Avg merge
3h 42m
Merged PRs (30d)
938

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from melodic-software/claude-code-plugins

All issues in melodic-software/claude-code-plugins

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.