vllm-project/semantic-router

feat: define signal model lifecycle, datasets, and release provenance

Open

#2,360 opened on Jul 5, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/researchenhancementenvironment/rocmevaluationhelp wantedpriority/P1roadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

Define the lifecycle for signal models and extractors, including datasets, evaluation, versioning, release provenance, and runtime compatibility.

Parent roadmap: #2287 Parent signal roadmap: #2335 Related: #1485, #2154, #2247, #2301, #2347, #2382 Related code: src/training, src/semantic-router/pkg/classification, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/modelinventory, candle-binding, nlp-binding, ml-binding

Motivation

H2 includes improving signal model accuracy and adding more extractors. The repo already has training/eval directories and runtime model bindings, but roadmap issues mostly describe individual models or bugs. Contributors need a lifecycle contract: what dataset/eval evidence is required, how model versions are published, how confidence calibration is measured, and how runtime compatibility is validated.

Scope

Define lifecycle requirements for:

  • training and evaluation datasets;
  • domain, complexity, modality, rerank, PII, feedback, hallucination, and custom signal extractors;
  • confidence calibration and threshold provenance;
  • model artifact versioning and catalog entries;
  • CPU/Candle/ONNX/native binding compatibility;
  • release notes and rollback behavior for signal model changes;
  • base-model and model-family research, publication, and migration requirements for next-generation router reranker, embedding, and classifier models (#2382).

Non-goals

  • Do not require every signal to use a learned model.
  • Do not accept accuracy improvements without reproducible eval evidence.
  • Do not silently change default thresholds without provenance.

Acceptance criteria

  • Signal model changes have dataset/eval/provenance requirements.
  • Runtime model catalogs can identify model version, capability, and compatibility constraints.
  • Accuracy and calibration reports are linked from relevant signal issues.
  • Default signal model updates include rollback guidance.

Contributor guide