feat(evaluator): handle engine-native interaction requests during evaluation
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- go
- Domain
- ai, testing-qa
Research direction
This is a design-level change to the evaluator's user simulator, so start by reading #266, #286 (shipped in v0.13.0) and the scenario-driven simulator they added, and note that #307 covers rule matching. The issue does not name specific files or entry points, so locate the simulator's between-turn message path first. Done means a versioned request/response contract, one engine adapter that answers a native request in the same session, and regression tests for the terminal reasons listed in the acceptance criteria.
Written by the indexing model from the issue text.
Description
Context
Follow-up to #266 and the scenario-driven user simulator implemented in #286 and released in v0.13.0.
The current simulator generates messages between completed Agent turns. It does not receive or answer engine-native approval/input requests while an Agent turn is still running. A question in natural-language output is also not a canonical indication that the Agent is waiting for input.
Proposed scope
Introduce a capability-based framework contract for structured interaction requests and responses, then implement and verify it for at least one engine that actually exposes such events.
- Distinguish completed turns from pending confirmation, choice, and information requests. Preserve the native request identity, prompt/options, and available context.
- Route a simulated response back to the pending request in the same running Agent session; do not silently start another session or treat an unresolved request as completion.
- Keep orchestration and termination in the evaluator. Define how in-turn interactions relate to the existing user-message/Agent-response turn count, and bound request count, timeouts, cancellation, and repeated/no-progress interactions.
- Constrain replies to scenario facts and permissions. Do not infer additional deployment or external-action authorization from the existence of an approval request.
- Record request/response provenance and explicit terminal reasons in artifacts and reports. The configured judge remains responsible for the final evaluation verdict.
- Preserve the existing between-turn simulator and scripted conversation behavior. Engines without the capability must have an explicit, documented unsupported path.
Acceptance criteria
- A versioned/explicit request-response contract distinguishes pending interaction from completed output.
- At least one real engine adapter handles a native request and response within the same session, with raw-event/artifact evidence.
- Completion, missing/stale request identity, unsupported engines, malformed replies, timeout, cancellation, and interaction limits have regression coverage.
- Scenario permission boundaries and request/response provenance are documented and tested.
- Legacy scripted and scenario-driven between-turn evaluations remain compatible.
Deterministic rule matching and optional LLM fallback are tracked in #307. Additional engine adapters can follow once the first adapter establishes the contract; this issue does not promise universal native approval support.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day