ci(evals): run the eval dispatch with TMPDIR outside /tmp so Bash cases can write

Closed Beginner friendly
  • Is something wrong?
#12,083 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, shell
Domain
ci-cd

Research direction

Start in the workflow file eval-browser-dispatch.yml, where the claude plugin eval step runs with the runner's default TMPDIR. Set TMPDIR on that step to a short directory outside /tmp (24 characters or fewer), then check the sandbox settings it produces. Done when a dispatch of a Bash-granting case that writes in its workspace shows no 'Read-only file system' line in its trace.

Written by the indexing model from the issue text.

Description

host-run priority: medium status: ready work-class: scoped

Refs: #12070

Outcome

eval-browser-dispatch.yml runs claude plugin eval with the runner's default TMPDIR, so every run root lands under /tmp. On Claude Code 2.1.316 the run's own sandbox settings then make that root read-only to Bash: sandbox.filesystem.allowWrite lists the run's home and tmp directories, and denyWrite lists //tmp, which covers both and wins. Any Bash-granting case that writes in its workspace (a git fetch, git merge, the verify record, the gh-forge stand-in's state) fails with Read-only file system in CI.

Evidence from the #12070 host run (2026-11-01, Claude Code 2.1.316, --runs 1 --ablation none, the two pull-request-ready-* cases):

  • Default TMPDIR: cannot lock ref 'ORIG_HEAD': Unable to create '<run>/home/cwd/.git/ORIG_HEAD.lock': Read-only file system, and zsh:2: read-only file system: <run>/tmp/claude-1000/cwd-*. Neither case reached gh pr ready.
  • TMPDIR outside /tmp: no read-only write in either trace, both cases reached gh pr ready and scored 1.0, validity gate VALID.

/evals:plugin-eval already tells local runs on Linux and WSL2 to set TMPDIR to a short directory outside the system temp directory.

Done when

  • The eval dispatch workflow runs claude plugin eval with TMPDIR set to a short directory outside /tmp (at most 24 characters, per /evals:plugin-eval's sandbox-start reference, so the sandbox's bridge sockets still fit).
  • A dispatch of a Bash-granting case that writes in its workspace shows no Read-only file system line in its trace.

Out of scope

  • Changing the eval cases or the gh-forge stand-in: with a writable root they need no change.
Dominant language
Shell
Stars
22
Forks
2
Avg merge
1h 40m
Merged PRs (30d)
918

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from melodic-software/claude-code-plugins

All issues in melodic-software/claude-code-plugins

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.