vllm-project/semantic-router

feature: optional two-stage cross-encoder rerank as a routing signal

Open

#2,251 opened on Jun 18, 2026

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/researchhelp wantedroadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Audited upstream baseline (2026-07-15)

Current main does not yet contain a general cross-encoder runtime or rerank endpoint. PR #2236 is open and unmerged. The existing router does have embedding/prototype scoring and bounded candidate sets that can provide the cheap first stage.

This issue begins after the typed model/runtime contract in #2250 is available. It must evaluate whether a cross-encoder improves route decisions; quality or calibration gains are hypotheses, not assumed production facts.

Scope

  1. Define an optional registered signal that reranks a bounded top-K candidate set.
  2. Keep the first-stage source explicit (for example a reviewed embedding shortlist).
  3. Emit typed score, margin, provenance, model version, latency, and fallback reason.
  4. Define timeout/invalid-output behavior that degrades to the reviewed first-stage result.
  5. Add decision-level offline/shadow evaluation for route correctness, latency, and cost.
  6. Keep the feature disabled by default until evaluation supports a preset.

Boundaries

  • Do not conflate document/KB reranking with model-route reranking; define the candidate and label semantics.
  • Do not call an unbounded set of candidates.
  • Do not block unrelated routing when the optional reranker is unavailable.
  • Do not bypass candidate authorization, safety policy, or the decision contract.

Acceptance criteria

  • The signal is optional and registered/observable through #2330.
  • Top-K and timeout limits are validated.
  • Missing, stale, or failed reranking has a typed fallback receipt.
  • Decision evaluation compares first-stage-only and two-stage routing on representative data.
  • Tests cover ranking, ties, timeout, invalid output, disabled state, and model unavailability.

Relevant surfaces

src/semantic-router/pkg/classification, typed signal/config registration, src/semantic-router/pkg/decision, model runtime (#2250), diagnostics, and #2247.

Contributor guide