Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Sandbox: the answer of /execute no longer says that a run was stopped at its time limit

Closed
  • Is something wrong?
#429 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, python
Domain
api, backend, testing

Research direction

Start with sandbox/executor_server.py and sandbox/resource_contract.py to trace /execute responses and _classify_error; check the size constraint on executor_server.py. Then read v3-service/pipeline.py and proxy/guardrails.go, and run the named timeout test plus plain-job tests for each resource limit. Done when /execute reports each limit, v3-service handles the error details, and the proxy gate test distinguishes a timed-out server from a red test.

Written by the indexing model from the issue text.

Description

area/sandbox reliability

What is wrong

A program that the sandbox stops at its time limit comes back from
/execute as a plain failure with no output: success is false,
error_type is null and error_message is empty.

Before 939b394 (2026-08-31, "one resource contract for every
untrusted command") such a run came back with the text "Execution
timed out after Ns" in stderr, so /execute answered
error_type: "Timeout" and a message with those words. Since that
commit stderr holds only what the program itself wrote. How the
run ended is in two new fields, timed_out and outcome. /shell
passes them on. /execute does not: its answer has no such field,
and its error_type is still made from stderr alone.

The limit itself acts as before: the program is stopped. What is
lost is the word about it. The commit is on dev only; no release
has it.

The test that shows it,
tests/infrastructure/test_sandbox.py::test_timeout_enforced_on_infinite_loop,
carries the integration mark and ran in no job. It failed on its
first run on a runner on 2026-10-08.

What needs doing

  1. The answer of /execute says how a run ended when the resource
    contract stopped it: for the time limit the type "Timeout" and a
    message again, taken from the contract's own fields and not from
    words in stderr. The same for the memory, process and output
    limits.
  2. Tests in the plain jobs hold it for each kind of limit.
  3. v3-service keeps only success, stdout and stderr of the
    answer; it never reads error_type. So it must read the new field
    too. Today a candidate that reached the time limit is sorted as an
    execution error, and the repair prompt gets an empty error text
    where it had "Execution timed out after Ns" before. The pull
    request shows both, before and after the change.
  4. The proxy's blockedServerStart (proxy/guardrails.go) looks for
    the words "execution timed out" in the result of a failed
    verification command, to tell a server that was started in the
    foreground from a red test. Nothing writes those words now. A test
    that builds the result as the proxy gets it today shows whether
    the gate still tells the two apart.

Why

A caller must be able to tell "stopped at the time limit" from
"failed with no output": the two need different next steps.

Done when

  • test_timeout_enforced_on_infinite_loop passes with no change
    to what it claims.
  • A test for each kind of limit runs in the plain jobs.
  • The pull request shows what v3-service sorts such a run into,
    and what error text its repair prompt gets, before and after.
  • A test holds the proxy's gate for a server that the time limit
    stopped, with the result as the proxy gets it today.

Where to start

  • sandbox/executor_server.py (the answer type of /execute,
    _classify_error), sandbox/resource_contract.py.
  • sandbox/executor_server.py is on the size list and may not grow.
  • v3-service/pipeline.py, the part that sorts a failed self-test.
Dominant language
Python
Stars
2.1k
Forks
197
Avg merge
14h 1m
Merged PRs (30d)
201

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from inferstep/ATLAS

All issues in inferstep/ATLAS

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.