vllm-project/semantic-router

feature: add tenant-aware entrypoint rules and caller-filtered model discovery

Ouverte

#2 868 ouverte le 11 août 2026

 (1 commentaire) (0 réaction) (0 personne assignée)Go (699 forks)github user discovery
area/corearea/dashboardarea/networkingenhancementhelp wantedpriority/P1safety

Métriques du dépôt

Stars
 (4 293 étoiles)
Métriques de merge PR
 (Métriques PR en attente)

Description

Describe the feature

Extend entrypoints from a static virtual-model-to-recipe mapping into a named, entrypoint-local rule table that can select a recipe from trusted request headers and the inference request path.

The same rules must be the source of truth for both caller-specific model discovery and invocation:

  • GET /v1/models lists only entrypoint aliases visible to the current caller.
  • Model-bearing inference requests resolve the requested alias, evaluate the entrypoint's local rules, and select exactly one recipe before any recipe-scoped signals or decisions run.
  • A caller that may use multiple entrypoints is represented by matching rules under each entrypoint; this proposal does not add a global entrypoint grant table.
  • Existing {model_names, recipe} entrypoints remain unconditional/public for compatibility.

The initial matcher surface should be intentionally small:

  • header Exact matching;
  • path Exact and segment-aware PathPrefix matching;
  • no method, query-parameter, regex, arbitrary body, or CEL matcher in v1.

Primary layer

global level

Why this layer?

Entrypoint resolution is a request boundary that runs before recipe-local signals, decisions, algorithms, and plugins. It affects the canonical config contract, OpenAI-compatible discovery, ExtProc request processing, control-plane APIs, and dashboard behavior, so it should be implemented as a dedicated pre-recipe routing capability rather than hidden inside a signal or plugin.

Why do you need this feature?

The current contract maps every entrypoint alias directly to one recipe and /v1/models publishes a static, unfiltered catalog. It cannot express cases such as:

  • tenant A using Auto through recipe A;
  • a specific user in tenant A using the same Auto alias through recipe B;
  • that user also being permitted to discover and invoke a separate Code entrypoint.

Using one local rule table for discovery and invocation prevents the model catalog and the actual routing boundary from drifting apart. It also prevents a conditional entrypoint miss from falling through to concrete-model passthrough or the default recipe.

Scope

  • Add stable entrypoint and rule names, rules[].matches[], header/path matchers, and recipe actions to the canonical config.
  • Add an immutable compiled resolver with explicit matched, claimed_no_match, unclaimed, and ambiguous outcomes.
  • Filter both ExtProc and router-API implementations of GET /v1/models through the same resolver.
  • Define caller visibility projection for collection discovery and an optional for_path query for endpoint-specific model discovery.
  • Add entrypoint-centric management CRUD, validation, and dry-run resolution APIs.
  • Update public model metadata so conditional entrypoints do not expose a misleading singular recipe.
  • Carry the contract through CLI validation, DSL round trips, Dashboard config/topology/playground flows, Helm/CRD/Operator translation, docs, conformance, and E2E.
  • Close direct-model, implicit-auto/looper, skip-processing, duplicate-header, caching, and ExtProc fail-open bypasses before describing the feature as a tenant authorization boundary.

Non-goals

  • Do not introduce a global user-to-entrypoint access-policy object.
  • Do not authenticate raw API keys in entrypoint matchers; authentication must validate credentials and inject trusted tenant/user/key-id claims first.
  • Do not replace recipe-local routing decisions or downstream model-selection algorithms.
  • Do not add weighted recipe actions or rule-scoped request mutation in the first version.

Acceptance criteria

  • Legacy {model_names, recipe} entrypoints retain their current public behavior.
  • Conditional entrypoints are visible and invocable only when a local rule matches the caller context.
  • /v1/models and invocation share one resolver and cannot disagree for the same caller and target path.
  • tenant=A,user=B deterministically outranks tenant=A without order-dependent behavior.
  • A claimed conditional alias with no matching rule is rejected and never becomes passthrough/default routing.
  • Unauthorized and nonexistent aliases use the same external 404 model_not_found response; an empty visible catalog remains 200 with data: [].
  • Conditional model-list entries omit singular recipe metadata and responses are not shared-cacheable across callers.
  • Duplicate or case-conflicting trusted identity headers fail closed.
  • Management CRUD uses ETag/If-Match, and recipe deletion accounts for every rule reference.
  • Direct backend IDs and implicit virtual aliases cannot bypass entrypoint isolation in strict multi-tenant mode.
  • Router, CLI, DSL, Dashboard, deployment translators, docs, conformance, and E2E preserve the same contract.

Related work

  • #2331 established multiple recipes and static entrypoints.
  • #2354 tracks normalization of auto and algorithm virtual aliases through entrypoints.
  • #2355 tracks Operator, CRD, and Helm parity for cross-surface config contracts.
  • #2801 covers service accounts, virtual keys, and trusted tenant context; those identities may feed these matchers, but this issue does not create another global grant model.
  • #1445 tracks spoofable identity headers when trusted authentication/header injection is absent.

Likely change surfaces

  • src/semantic-router/pkg/config
  • src/semantic-router/pkg/extproc
  • src/semantic-router/pkg/publicmodels
  • src/semantic-router/pkg/apiserver
  • src/semantic-router/pkg/services
  • src/semantic-router/pkg/dsl
  • src/vllm-sr/cli
  • dashboard
  • deploy/helm and deploy/operator
  • e2e, config/recipes, and public documentation

The first comment contains the proposed config schema, exact matching semantics, API behavior, management permissions, security boundaries, and implementation/validation plan.

Guide contributeur