OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remote

Open
#420 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@bababoom721 is already working on this.

Since Nov 17, 2026.

  • #421 by @bababoom721 — closed without merging
  • #424 by @bababoom721 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
docker, go

Research direction

Start by reading judge.fileExistsInWorkspace and tracing how runtime workspace paths reach judge.Input.WorkspacePath for expect and rule_based checks. Inspect how OpenSandbox and Docker download workspace files, then run the relevant file-check tests with a remote runtime. Done means checks find files staged locally while judging remains outside the sandbox.

Written by the indexing model from the issue text.

Description

Problem

When environment.type: opensandbox is used, expect.files_exist and expect.file_contains always fail even though the Agent creates the expected files in the sandbox workspace.

Reproduction

  1. Run an eval with environment.type: opensandbox.
  2. Have the Agent write findings.json to /workspace.
  3. Configure:
expect:
  files_exist: ["findings.json"]

Observed: collect_artifacts downloads findings.json, but grading still reports expected file "findings.json" does not exist.

Root cause

judge.fileExistsInWorkspace calls os.Stat on rt.Workspace(). For OpenSandbox and Docker runtimes, rt.Workspace() is the remote container path such as /workspace, which does not exist on the host filesystem. Only the none runtime has a host-accessible workspace.

Expected behavior

The evaluated Agent should run inside the sandbox, while judging stays outside the sandbox. File-based expect checks should first materialize the remote workspace or the referenced files into a local staging directory, then run the existing local checks.

Proposed direction

For runtimes that do not own a host workspace, download the workspace to a temporary local directory before expect and rule_based file checks, and point judge.Input.WorkspacePath at that local directory.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.