vllm-project/semantic-router

feature: carry compute policy and routing evidence into Looper execution

オープン

#2,861 opened on 2026/08/11

 (1 件のコメント) (0 件のリアクション) (0 人の担当者)Go (699 件のフォーク)github user discovery
area/corearea/model-selectionarea/momenhancementhelp wantedpriority/P1roadmap

Repository metrics

Stars
 (4,293 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Summary

Introduce request-scoped compute-policy and routing-evidence contracts for Looper execution so algorithms can make bounded, explainable decisions using the Router's existing signals, decision confidence, model pricing, and remaining budget.

Parent: #2336 Hard call-amplification safety floor: #1456 Related attempt accounting: #2855 Related fixed-budget evaluation: #2858

Motivation

looper.Request currently receives candidate models and a static algorithm config, but not the evidence that selected the decision or a cross-stage budget ledger. Complexity, fact-check, context, and decision confidence are available in the request context before Looper execution, while model pricing is available in ModelParams, yet algorithms cannot use those facts to allocate or stop work consistently.

Concurrency and per-call token limits are not a complete request budget. #1456 remains the canonical owner of the hard total-call/amplification cap; this issue adds the policy/evidence layer above that invariant.

Proposed contracts

  • RoutingEvidence: selected-decision confidence plus bounded matched signal names/values needed by an algorithm, without prompt content.
  • ComputeBudget: maximum prompt/completion/total tokens, estimated cost, and wall time; integrate the hard call limit provided by #1456 rather than duplicating it.
  • BudgetLedger: attempted/committed/consumed resources and a typed exhaustion reason available before each new stage or candidate batch.

Scope

  • Extend the Looper request construction path with bounded routing evidence.
  • Define token, cost, and wall-time budget semantics, defaults, validation, and precedence.
  • Estimate cost from configured model pricing and update the ledger from actual backend usage when available.
  • Define deterministic behavior when a budget is absent, exhausted, or cannot be estimated.
  • Make the ledger consumable by existing algorithms without implementing adaptive-N or changing default schedules in this issue.
  • Integrate with #1456's hard call counter as a non-overridable safety constraint.

Non-goals

  • Do not duplicate the total upstream-call cap tracked in #1456.
  • Do not add adaptive-N, Best-of-N, or cross-algorithm chaining.
  • Do not expose raw prompts, response bodies, or sensitive signal payloads.
  • Do not make batch/capacity-aware global scheduling part of per-query routing.

Acceptance criteria

  • Looper execution can read decision confidence and declared routing evidence from a typed request contract.
  • Algorithms can query remaining token, cost, wall-time, and hard-call budgets before starting more work.
  • Actual usage updates the ledger; missing usage has documented conservative behavior.
  • Model pricing drives estimated cost with explicit handling for unknown/zero pricing.
  • Exhaustion produces a stable reason and deterministic fail/fallback policy.
  • #1456's hard call limit cannot be raised or bypassed by the policy layer.
  • Replay/debug records the chosen budget and final consumption without sensitive content.

Validation

  • Contract, precedence, and exhaustion unit tests.
  • Missing-usage and missing-pricing tests.
  • ExtProc-to-Looper routing-evidence integration tests.
  • Exact-boundary tests composed with #1456.

コントリビューターガイド