Eval driver: a per-suite session budget, the same for both arms
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- ai-infra-agents
Research direction
Start by tracing how the eval stack configures suites and passes session limits to the ATLAS and baseline arms, then find where evaluation records are written. Ensure the suite's time and turn budgets reach both arms and are reflected in the record; done means the paired arms use the same suite budget and the record shows it.
Written by the indexing model from the issue text.
Description
What needs doing
Set the session time and turn budget per suite on the eval stack, and
apply the same budget to the ATLAS arm and the baseline arm.
Why
Some benchmarks define their own time limits, often above 600 s. A
budget that differs between the arms would bias the paired comparison.
Done when
- The suite's budget reaches both arms, and the record shows it.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 197
- Avg merge
- 5h 13m
- Merged PRs (30d)
- 604
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from inferstep/ATLAS
-
help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Bootstrap: the last two steps run as the user who owns the installMay be free again @assassinationss claimed this 27 days ago, and no pull request is open. Openarea/install reliability
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
inferstep/ATLAS#412 · 4 comments ·
Maintainers usually reply within 1 day
-
area/cli
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
area/cli area/lens status/ready
Difficulty 2/5 Half a day Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
area/ci status/ready
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
inferstep/ATLAS#245 · 4 comments ·
Maintainers usually reply within 1 day
Similar issues
-
bug severity:low
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 2 days
-
curriculum documentation quality
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
githubnext/gh-aw-workshop#5367 ·
Maintainers usually reply within 3 days
-
OAuth documentation for GET /api/v2/users/me/ appears inconsistent with required global read scopeOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Create Noise Latents ignores randomize_seed and produces the same noise every runPossibly taken @griptapeops claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
griptape-ai/griptape-nodes-library-diffusers#127 ·
Maintainers usually reply within 1 day
-
Jenkins capacity report: metrics history is lost when archiving a month failsPossibly taken @RaphaelFakhri claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
adoptium/infrastructure#4568 ·
Maintainers usually reply within 1 day