vllm-project/semantic-router

feature: define a pluggable Looper outcome-verifier contract

Ouverte

#2 857 ouverte le 11 août 2026

 (1 commentaire) (0 réaction) (0 personne assignée)Go (699 forks)github user discovery
area/corearea/model-selectionarea/momenhancementhelp wantedpriority/P1roadmap

Métriques du dépôt

Stars
 (4 293 étoiles)
Métriques de merge PR
 (Métriques PR en attente)

Description

Summary

Define a shared outcome-verifier contract for Looper algorithms so candidate scoring, acceptance, reranking, and verifier diagnostics do not remain tied to one algorithm.

Parent: #2336 Related research tracker: #2337 Related attempt trace: #2855

Current behavior

Verification is fragmented:

  • confidence owns AutoMix HTTP entailment and inline self-verification;
  • Fusion owns NLI/hallucination grounding function hooks;
  • Workflows expresses verifier/reviewer/critic only as prompt roles;
  • no common interface reports a calibrated score, decision, cost, or trace.

Panel consistency is also not truth: the existing Grounding-aware Fusion evaluation shows that hard filtering can remove a correct minority answer.

Proposed contract

A verifier should accept the original task, one or more normalized candidates, and optional trusted references/context, then return:

  • scalar score and optional calibrated acceptance decision;
  • stable verifier type/version;
  • bounded evidence/reason codes;
  • usage and latency when the verifier itself consumes model compute;
  • typed unavailable/timeout/failure behavior.

Initial adapter families:

  • deterministic checker or executable result;
  • HTTP outcome/process reward service;
  • LLM scalar/reranking judge;
  • LLM pairwise comparator;
  • existing AutoMix and reusable grounding signals where their semantics fit.

Scope

  • Add the shared interface and result/trace types.
  • Adapt confidence AutoMix without changing its externally visible behavior.
  • Provide at least one deterministic test verifier and one model/HTTP adapter.
  • Define calibration, timeout, and on_error semantics.
  • Integrate verifier usage with common attempt tracing.

Non-goals

  • Do not claim NLI agreement is a correctness oracle.
  • Do not add Best-of-N or verifier-driven retry loops in this issue.
  • Do not implement token-level PRM beam search in the Go router.

Acceptance criteria

  • At least two Looper paths can consume the same verifier contract.
  • Verifier latency, usage, score, version, and decision are traceable.
  • Timeout/unavailable/error semantics are deterministic and tested.
  • Trusted context is explicit and cannot be confused with peer agreement.
  • Existing confidence AutoMix behavior remains compatible.
  • The contract can support future outcome reranking without another algorithm-specific interface.

Validation

  • Contract and adapter unit tests.
  • Confidence integration regression tests.
  • Failure/timeout and sensitive-data redaction tests.

Guide contributeur