feat(user-simulator): add deterministic response rules with optional LLM fallback

Open
#307 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
testing-qa

Research direction

Start from the existing LLM user simulator introduced in #286 and the interaction request contract being defined in #306, since this issue depends on both. Design the rule schema, operators, precedence and ambiguity handling before writing any engine code. Done means documented and validated rules, reproducible matches without an LLM call, explicit unmatched and fallback behavior, and a runnable example.

Written by the indexing model from the issue text.

Description

enhancement

Context

Follow-up to #266 and the initial LLM user simulator implemented in #286 and released in v0.13.0.

The current implementation uses a constrained LLM to produce dynamic user replies. Exact confirmation/choice/information requests would benefit from reproducible rule-based responses and an explicitly configured LLM fallback for unmatched requests.

Proposed scope

Add deterministic response rules over a stable structured interaction request contract. Define that contract with #306 rather than making natural-language classification by another LLM appear deterministic.

  • Match explicit request type, identity, options, and documented fields using deterministic operators; define rule ordering, precedence, and ambiguous-match behavior.
  • Allow rules to select an option or supply a bounded reply from declared scenario facts and permissions.
  • Make unmatched behavior explicit: stop/error or an opt-in constrained LLM fallback. Do not silently expand permissions or invent missing facts.
  • Record which rule matched, whether fallback was used, and the resulting response and terminal reason.
  • Share evaluator-owned timeout/cancellation and interaction/turn limits with the existing orchestration. Replies do not determine PASS/FAIL; judges still do.

Acceptance criteria

  • Rule schema, matching fields/operators, precedence, and ambiguity behavior are documented and validated.
  • Matching rules produce reproducible replies without an LLM call.
  • Unmatched requests and optional LLM fallback have explicit, tested behavior.
  • Unknown facts, forbidden approvals, malformed responses, cancellation, and budget exhaustion are covered.
  • Artifacts identify the matched rule or fallback and preserve request/response evidence.
  • A runnable example demonstrates rules with fallback and preserves existing LLM-only simulator behavior.

Native in-process engine event handling is tracked in #306. If rules are also offered for completed-turn output, their input normalization and limitations must be explicitly defined rather than inferred from list order or free text.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.