plugin eval: an Agent call with `isolation: "worktree"` is refused, and the plugin-evals page doesn't say so

Open Beginner friendly
  • Is something wrong?
#108,157 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
git, typescript
Domain
cli, developer-experience

Research direction

Start with the plugin-evals documentation and its § How runs are isolated section, then check claude plugin eval --help to confirm the current behavior. The issue gives a clear close condition: either worktree isolation works in eval runs or the documentation explicitly says it is refused. Update the documentation to match the behavior you verify.

Written by the indexing model from the issue text.

Description

area:agents area:plugins bug has repro platform:macos
What happens

In a claude plugin eval run, an Agent tool call that sets isolation: "worktree" fails with:

Cannot create agent worktree: worktree isolation is unavailable in this session (plugin evaluation harness)

The same call succeeds in a claude -p session in a repository of the same shape: the subagent runs in its own worktree under .claude/worktrees/.

Why it matters

A skill that dispatches a subagent into its own worktree, so the subagent can edit and run the project's code without writing the user's checkout, can't have that path measured with claude plugin eval: every case that reaches the dispatch scores the refusal instead. The plugin-evals page's § How runs are isolated lists what a run turns off, such as the Artifact tool, and doesn't mention worktree isolation, and claude plugin eval --help names no option for it, so a case author finds out from a failed run.

Reproducer

A plugin with nothing but a manifest and one case:

.claude-plugin/plugin.json

{
  "name": "repro-isolation",
  "version": "0.0.1",
  "description": "Reproducer: an Agent call with isolation worktree inside claude plugin eval."
}

evals/isolated-subagent/prompt.md

---
expected_outcome: "The subagent runs in its own worktree and replies with the output of `git rev-parse --show-toplevel`."
max_turns: 10
allowed_tools: [Agent, Bash, Read]
---

The working directory is a git repository with one commit. Use the Agent tool once: dispatch a general-purpose subagent with `isolation: "worktree"`, asking it to run `git rev-parse --show-toplevel` and reply with the output. Then report the subagent's reply, or the exact error the Agent tool returned.

evals/isolated-subagent/case.yaml

schema_version: "1.1"
name: isolated-subagent
context:
  scaffold_script: scaffold.sh

evals/isolated-subagent/scaffold.sh

#!/usr/bin/env bash
set -euo pipefail
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
git init --quiet --initial-branch=main .
echo hello > README.md
git add README.md
git -c user.name=Example -c [email protected] commit --quiet -m "One file"

evals/isolated-subagent/graders/isolated.md

---
type: tool_used
tool: Agent
input_match: '"isolation"\s*:\s*"worktree"'
---

Run claude plugin eval . --scaffold --allow-tools Bash --runs 1 --keep-temp. The session makes the one Agent call, gets the error above as its tool result, and reports it; the subagent never starts.

For comparison, build the same repository with scaffold.sh in an empty directory and run the prompt's body there with claude -p "<prompt>" --allowedTools Agent Bash Read --permission-mode dontAsk. The subagent runs, and git rev-parse --show-toplevel prints <repository>/.claude/worktrees/agent-<id>.

Claude Code 2.1.319, macOS 26.5.1.

Suggested close condition

Either an Agent call with isolation: "worktree" works in an eval run (its worktree inside the run's workspace, where the run's sandbox already lets Bash write), or plugin-evals § How runs are isolated says such a call is refused, as it says the Artifact tool is off.

Dominant language
TypeScript
Stars
154k
Forks
25.6k
Avg merge
12h 27m
Merged PRs (30d)
4

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from anthropics/claude-code

All issues in anthropics/claude-code

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.