feat(user-simulator): add deterministic response rules with optional LLM fallback
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go
- Domain
- testing-qa
Research direction
Start from the existing LLM user simulator introduced in #286 and the interaction request contract being defined in #306, since this issue depends on both. Design the rule schema, operators, precedence and ambiguity handling before writing any engine code. Done means documented and validated rules, reproducible matches without an LLM call, explicit unmatched and fallback behavior, and a runnable example.
Written by the indexing model from the issue text.
Description
Context
Follow-up to #266 and the initial LLM user simulator implemented in #286 and released in v0.13.0.
The current implementation uses a constrained LLM to produce dynamic user replies. Exact confirmation/choice/information requests would benefit from reproducible rule-based responses and an explicitly configured LLM fallback for unmatched requests.
Proposed scope
Add deterministic response rules over a stable structured interaction request contract. Define that contract with #306 rather than making natural-language classification by another LLM appear deterministic.
- Match explicit request type, identity, options, and documented fields using deterministic operators; define rule ordering, precedence, and ambiguous-match behavior.
- Allow rules to select an option or supply a bounded reply from declared scenario facts and permissions.
- Make unmatched behavior explicit: stop/error or an opt-in constrained LLM fallback. Do not silently expand permissions or invent missing facts.
- Record which rule matched, whether fallback was used, and the resulting response and terminal reason.
- Share evaluator-owned timeout/cancellation and interaction/turn limits with the existing orchestration. Replies do not determine PASS/FAIL; judges still do.
Acceptance criteria
- Rule schema, matching fields/operators, precedence, and ambiguity behavior are documented and validated.
- Matching rules produce reproducible replies without an LLM call.
- Unmatched requests and optional LLM fallback have explicit, tested behavior.
- Unknown facts, forbidden approvals, malformed responses, cancellation, and budget exhaustion are covered.
- Artifacts identify the matched rule or fallback and preserve request/response evidence.
- A runnable example demonstrates rules with fallback and preserves existing LLM-only simulator behavior.
Native in-process engine event handling is tracked in #306. If rules are also offered for completed-turn output, their input normalization and limitations must be explicitly defined rather than inferred from list order or free text.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day