Eval driver: run a task's session and grader in a task-supplied container image
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
Research direction
Start by reading docs/EVAL_INTERFACE.md and tracing the eval driver in scripts/eval/ to see how it selects the sandbox image and runs session commands and graders. Then check how ATLAS's run_command is wired into those sessions. Done means a toy suite using two task-specific images runs both arms end to end, and run_command executes in the task image.
Written by the indexing model from the issue text.
Description
What needs doing
Let the eval driver (scripts/eval/, docs/EVAL_INTERFACE.md) take a
container image per task, and run both the ATLAS session's commands and
the grader in it, with the workspace at the repository root. Today the
driver assumes the one ATLAS sandbox image.
Why
Repository-level public benchmarks ship one environment per task. The
public comparison runs after the held-out proofs (#238).
Done when
- A toy suite with two different task images runs both arms end to end.
- ATLAS's
run_commandexecutes in the task image during those sessions.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 197
- Avg merge
- 14h 1m
- Merged PRs (30d)
- 201
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from inferstep/ATLAS
-
Bootstrap: the last two steps run as the user who owns the installMay be free again @assassinationss claimed this 18 days ago, and no pull request is open. Openarea/install reliability
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
inferstep/ATLAS#412 · 4 comments ·
Maintainers usually reply within 1 day
-
area/cli
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
area/cli area/lens status/ready
Difficulty 2/5 Half a day Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
area/ci status/ready
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
inferstep/ATLAS#245 · 4 comments ·
Maintainers usually reply within 1 day
-
area/docs documentation help wanted
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 1 day
Similar issues
-
bug P1
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Toloka/tolokaforge#1947 ·
Maintainers usually reply within 1 day
-
SageMaker provider: streaming fails with NoStreamDataError on DJL Serving (LMI) endpointsPossibly taken @balajiaru claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
UKGovernmentBEIS/inspect_ai#6127 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
pydot/pydot#601 · 4 comments ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
letsencrypt/cp-cps#361 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100