feat(user-simulator): add deterministic response rules with optional LLM fallback

Aperta
#307 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
testing-qa

Direzione di ricerca

Start from the existing LLM user simulator introduced in #286 and the interaction request contract being defined in #306, since this issue depends on both. Design the rule schema, operators, precedence and ambiguity handling before writing any engine code. Done means documented and validated rules, reproducible matches without an LLM call, explicit unmatched and fallback behavior, and a runnable example.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Context

Follow-up to #266 and the initial LLM user simulator implemented in #286 and released in v0.13.0.

The current implementation uses a constrained LLM to produce dynamic user replies. Exact confirmation/choice/information requests would benefit from reproducible rule-based responses and an explicitly configured LLM fallback for unmatched requests.

Proposed scope

Add deterministic response rules over a stable structured interaction request contract. Define that contract with #306 rather than making natural-language classification by another LLM appear deterministic.

  • Match explicit request type, identity, options, and documented fields using deterministic operators; define rule ordering, precedence, and ambiguous-match behavior.
  • Allow rules to select an option or supply a bounded reply from declared scenario facts and permissions.
  • Make unmatched behavior explicit: stop/error or an opt-in constrained LLM fallback. Do not silently expand permissions or invent missing facts.
  • Record which rule matched, whether fallback was used, and the resulting response and terminal reason.
  • Share evaluator-owned timeout/cancellation and interaction/turn limits with the existing orchestration. Replies do not determine PASS/FAIL; judges still do.

Acceptance criteria

  • Rule schema, matching fields/operators, precedence, and ambiguity behavior are documented and validated.
  • Matching rules produce reproducible replies without an LLM call.
  • Unmatched requests and optional LLM fallback have explicit, tested behavior.
  • Unknown facts, forbidden approvals, malformed responses, cancellation, and budget exhaustion are covered.
  • Artifacts identify the matched rule or fallback and preserve request/response evidence.
  • A runnable example demonstrates rules with fallback and preserves existing LLM-only simulator behavior.

Native in-process engine event handling is tracked in #306. If rules are also offered for completed-turn output, their input normalization and limitations must be explicitly defined rather than inferred from list order or free text.

Lingua principale
Go
Stelle
1.5k
Fork
134
Merge medio
1g 22h
PR unite (30g)
61

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di alibaba/skill-up

Tutte le issue di alibaba/skill-up

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.