feat(evaluator): handle engine-native interaction requests during evaluation

Open
#306 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
go
Domain
ai, testing-qa

Research direction

This is a design-level change to the evaluator's user simulator, so start by reading #266, #286 (shipped in v0.13.0) and the scenario-driven simulator they added, and note that #307 covers rule matching. The issue does not name specific files or entry points, so locate the simulator's between-turn message path first. Done means a versioned request/response contract, one engine adapter that answers a native request in the same session, and regression tests for the terminal reasons listed in the acceptance criteria.

Written by the indexing model from the issue text.

Description

enhancement

Context

Follow-up to #266 and the scenario-driven user simulator implemented in #286 and released in v0.13.0.

The current simulator generates messages between completed Agent turns. It does not receive or answer engine-native approval/input requests while an Agent turn is still running. A question in natural-language output is also not a canonical indication that the Agent is waiting for input.

Proposed scope

Introduce a capability-based framework contract for structured interaction requests and responses, then implement and verify it for at least one engine that actually exposes such events.

  • Distinguish completed turns from pending confirmation, choice, and information requests. Preserve the native request identity, prompt/options, and available context.
  • Route a simulated response back to the pending request in the same running Agent session; do not silently start another session or treat an unresolved request as completion.
  • Keep orchestration and termination in the evaluator. Define how in-turn interactions relate to the existing user-message/Agent-response turn count, and bound request count, timeouts, cancellation, and repeated/no-progress interactions.
  • Constrain replies to scenario facts and permissions. Do not infer additional deployment or external-action authorization from the existence of an approval request.
  • Record request/response provenance and explicit terminal reasons in artifacts and reports. The configured judge remains responsible for the final evaluation verdict.
  • Preserve the existing between-turn simulator and scripted conversation behavior. Engines without the capability must have an explicit, documented unsupported path.

Acceptance criteria

  • A versioned/explicit request-response contract distinguishes pending interaction from completed output.
  • At least one real engine adapter handles a native request and response within the same session, with raw-event/artifact evidence.
  • Completion, missing/stale request identity, unsupported engines, malformed replies, timeout, cancellation, and interaction limits have regression coverage.
  • Scenario permission boundaries and request/response provenance are documented and tested.
  • Legacy scripted and scenario-driven between-turn evaluations remain compatible.

Deterministic rule matching and optional LLM fallback are tracked in #307. Additional engine adapters can follow once the first adapter establishes the contract; this issue does not promise universal native approval support.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.