After the rework: re-run the frozen held-out suite to show nothing changed
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- ai-infra-agents, python
- Domain
- testing-qa
Research direction
First locate the frozen held-out suite and the earlier run's aggregates; the issue does not name their files or commands. Run the suite after the rework and compare aggregate results with the prior run. Done means the aggregates match within their intervals, or every difference has an explanation.
Written by the indexing model from the issue text.
Description
What needs doing
Once the rework lands, run the frozen held-out suite again and compare
the aggregates with the run before it.
Why
Unit, e2e and replay tests check pieces and recorded sessions. The
held-out suite checks that the whole product still behaves the same
with a real model after a large rework.
Done when
- The aggregates match the earlier run within their intervals, or
each difference has an explanation.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 197
- Avg merge
- 12h 11m
- Merged PRs (30d)
- 146
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from inferstep/ATLAS
-
Bootstrap: the last two steps run as the user who owns the installMay be free again @assassinationss claimed this 8 days ago, and no pull request is open. Openarea/install reliability
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
inferstep/ATLAS#412 · 4 comments ·
Maintainers usually reply within 1 day
-
Installer and setup guide: say what the docker group givesPossibly taken @se1vapriya claimed this 9 days ago. Openarea/install documentation good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
inferstep/ATLAS#411 · 5 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
area/cli
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
A `sandbox_subdir` the proxy does not accept is refused, not ignoredPossibly taken @mvanhorn claimed this 4 days ago. Openarea/proxy reliability
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
inferstep/ATLAS#364 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
area/cli area/lens status/ready
Difficulty 2/5 Half a day Newbie friendliness 66/100
Maintainers usually reply within 1 day
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Graphify-Labs/graphify#4524 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
topoteretes/cognee#5982 ·
Maintainers usually reply within 1 day
-
Patches of derivations with `__structuredAttrs` are ignoredPossibly taken @KiaraGrouwstra claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 81/100
nix-community/vulnix#184 ·
-
New InternshipOpennew_internship
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
aws-samples/sample-per-model-bedrock#295 ·
Maintainers usually reply within 1 day