feat(runtime): option to isolate local runs from the user's own agent setup

Open
#404 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
devtools

Research direction

Start in internal/runtime/runtime.go, where mergeEnv builds the environment for the none runtime from os.Environ(), then read internal/agent/claude_code.go to see how the Skill is installed into the workspace. Agree on the setting name and the per-case temporary home with the maintainers before coding, since the proposal leaves the design open. Done means an opt-in setting gives each case its own HOME and engine config directory, removed with the workspace unless --no-delete is set.

Written by the indexing model from the issue text.

Description

Problem / motivation

On environment.type: none, every agent command inherits the host environment: mergeEnv starts from os.Environ() (runtime.go#L31). So the agent runs with the user's real home directory and loads the user's own setup into the eval: settings, MCP servers, memory files such as ~/.claude/CLAUDE.md or ~/.codex/AGENTS.md, and user-level Skills.

That can make the with/without comparison wrong:

  1. The without_skill arm can still have the Skill. A Skill author often has the Skill installed globally while developing it, and the README installs skill-upper globally (npx skills add ... -g). A user-level copy is visible in both arms.
  2. The with_skill arm can run a different copy. skill-up installs the Skill under test into the workspace's .claude/skills (claude_code.go#L43). Claude Code's docs say a personal Skill wins over a project Skill with the same name: "With deploy in both ~/.claude/skills/ and the project's .claude/skills/, /deploy runs the personal one" (Claude Code docs). So an edited copy under evaluation can lose to an older installed copy.
  3. Results depend on whose machine ran them, and runs write into the user's own agent state (#360, #366).

I have not reproduced point 2 with a model run. It follows from the install path and the Claude Code docs.

Users already work around this by hand:

  • powercontext's eval README: "Use a dedicated evaluation account and profile: Claude inherits user settings, native memory, and global Skills/MCP." (README)
  • I ran my own evals with a temporary HOME for the same reason.

Proposed solution

An opt-in setting for the none runtime, for example environment.isolate_home: true (name open). For each case it would:

  1. Create a temporary home directory next to the case workspace, and point HOME and the engine's own config variable at it (CLAUDE_CONFIG_DIR, CODEX_HOME, and so on).
  2. Carry over only what the engine needs to log in. API keys in the environment pass through as today. Each adapter decides which credential files to copy or link.
  3. Remove it together with the workspace, keeping it under --no-delete.

Docker and OpenSandbox already start from a clean home, so they would not change.

Login storage differs per engine: some keep it in files under the home directory, others in the OS keychain. An engine whose login cannot be carried over should fail early with a clear message instead of running logged out. I can start with Claude Code and Codex, which I can test.

Alternatives considered

  • Run in Docker or OpenSandbox. That isolates, but it is heavier, and the docs recommend none for plain-text Skills.
  • Clean up after the run (#394, #397). That removes leftovers, but the run itself still reads the user's setup.
  • Tell users to use a separate OS account. That works, but it puts the setup on every user.

Additional context

  • #267 (closed as not planned) was about cleaning up after a run. This issue is about what the agent reads during the run.
  • #360 and PR #398 stop the Codex adapter from writing mocked servers into the user's global config.
Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.