OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remote
Maintainers usually reply within 1 day
@bababoom721 is already working on this.
Since Nov 17, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- docker, go
- Domain
- ai-infra-agents, testing
Research direction
Start by reading judge.fileExistsInWorkspace and tracing how runtime workspace paths reach judge.Input.WorkspacePath for expect and rule_based checks. Inspect how OpenSandbox and Docker download workspace files, then run the relevant file-check tests with a remote runtime. Done means checks find files staged locally while judging remains outside the sandbox.
Written by the indexing model from the issue text.
Description
Problem
When environment.type: opensandbox is used, expect.files_exist and expect.file_contains always fail even though the Agent creates the expected files in the sandbox workspace.
Reproduction
- Run an eval with
environment.type: opensandbox. - Have the Agent write
findings.jsonto/workspace. - Configure:
expect:
files_exist: ["findings.json"]
Observed: collect_artifacts downloads findings.json, but grading still reports expected file "findings.json" does not exist.
Root cause
judge.fileExistsInWorkspace calls os.Stat on rt.Workspace(). For OpenSandbox and Docker runtimes, rt.Workspace() is the remote container path such as /workspace, which does not exist on the host filesystem. Only the none runtime has a host-accessible workspace.
Expected behavior
The evaluated Agent should run inside the sandbox, while judging stays outside the sandbox. File-based expect checks should first materialize the remote workspace or the referenced files into a local staging directory, then run the existing local checks.
Proposed direction
For runtimes that do not own a host workspace, download the workspace to a temporary local directory before expect and rule_based file checks, and point judge.Input.WorkspacePath at that local directory.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#405 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day