vllm-project/semantic-router
feature: define a pluggable Looper outcome-verifier contract
Open
#2,857 opened on Aug 11, 2026
area/corearea/model-selectionarea/momenhancementhelp wantedpriority/P1roadmap
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Define a shared outcome-verifier contract for Looper algorithms so candidate scoring, acceptance, reranking, and verifier diagnostics do not remain tied to one algorithm.
Parent: #2336 Related research tracker: #2337 Related attempt trace: #2855
Current behavior
Verification is fragmented:
- confidence owns AutoMix HTTP entailment and inline self-verification;
- Fusion owns NLI/hallucination grounding function hooks;
- Workflows expresses verifier/reviewer/critic only as prompt roles;
- no common interface reports a calibrated score, decision, cost, or trace.
Panel consistency is also not truth: the existing Grounding-aware Fusion evaluation shows that hard filtering can remove a correct minority answer.
Proposed contract
A verifier should accept the original task, one or more normalized candidates, and optional trusted references/context, then return:
- scalar score and optional calibrated acceptance decision;
- stable verifier type/version;
- bounded evidence/reason codes;
- usage and latency when the verifier itself consumes model compute;
- typed unavailable/timeout/failure behavior.
Initial adapter families:
- deterministic checker or executable result;
- HTTP outcome/process reward service;
- LLM scalar/reranking judge;
- LLM pairwise comparator;
- existing AutoMix and reusable grounding signals where their semantics fit.
Scope
- Add the shared interface and result/trace types.
- Adapt confidence AutoMix without changing its externally visible behavior.
- Provide at least one deterministic test verifier and one model/HTTP adapter.
- Define calibration, timeout, and
on_errorsemantics. - Integrate verifier usage with common attempt tracing.
Non-goals
- Do not claim NLI agreement is a correctness oracle.
- Do not add Best-of-N or verifier-driven retry loops in this issue.
- Do not implement token-level PRM beam search in the Go router.
Acceptance criteria
- At least two Looper paths can consume the same verifier contract.
- Verifier latency, usage, score, version, and decision are traceable.
- Timeout/unavailable/error semantics are deterministic and tested.
- Trusted context is explicit and cannot be confused with peer agreement.
- Existing confidence AutoMix behavior remains compatible.
- The contract can support future outcome reranking without another algorithm-specific interface.
Validation
- Contract and adapter unit tests.
- Confidence integration regression tests.
- Failure/timeout and sensitive-data redaction tests.