CI: replay tests that run the proxy against recorded model replies
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 50/100
Research direction
Start by locating the proxy's loop, existing tests, and CI configuration; the issue does not name specific files. Check how the proxy handles model replies and how CI runs its tests, then build replay coverage for a normal session and malformed, cut-off, and looping replies. Done means the replays compare tool calls, refusals, and final status, and run in CI on pull requests that touch the proxy.
Written by the indexing model from the issue text.
Description
What needs doing
A test harness with a stand-in model server that plays back recorded
replies, including the bad ones: a malformed tool call, a reply that
is cut off, a reply that loops. The proxy runs its real loop against
it, and the test compares what the proxy did (tool calls, refusals,
the final status) with a recorded expectation.
Recordings come from smoke, development and end-to-end tasks only.
A change to a recording needs a maintainer's approval and a stated
reason, because a recording is an expected value.
Why
The proxy is about to be restructured in many small steps. Unit tests
check pieces. A replay shows that the whole loop still behaves the
same after each step, without a GPU and in seconds.
Done when
- The harness runs in CI on every pull request that touches the
proxy. - It covers a normal session and at least the three bad-reply
cases above. - Changing loop behaviour on purpose makes a replay test fail.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 197
- Avg merge
- 14h 1m
- Merged PRs (30d)
- 201
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from inferstep/ATLAS
-
Bootstrap: the last two steps run as the user who owns the installMay be free again @assassinationss claimed this 16 days ago, and no pull request is open. Openarea/install reliability
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
inferstep/ATLAS#412 · 4 comments ·
Maintainers usually reply within 1 day
-
area/cli
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
area/cli area/lens status/ready
Difficulty 2/5 Half a day Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
area/ci status/ready
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
inferstep/ATLAS#245 · 4 comments ·
Maintainers usually reply within 1 day
-
area/docs documentation help wanted
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
google-agentic-commerce/AP2#388 ·
-
area:codegen backend:cuda backend:hip bug direction:input
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
CrossGL/crosstl#3061 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
rlaope/oh-my-hermes#2304 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
mochan-tk/agentic-dev-kit-for-codex#64 ·
Maintainers usually reply within 1 day
-
bug docs performance
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 3 days