vllm-project/semantic-router

feat: define signal model lifecycle, datasets, and release provenance

Offen

#2.360 geöffnet am 05.07.2026

 (0 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Go (699 Forks)github user discovery
area/researchenhancementenvironment/rocmevaluationhelp wantedpriority/P1roadmapsignal-decision-engine

Repository-Metriken

Stars
 (4.293 Sterne)
PR-Merge-Metriken
 (PR-Metriken ausstehend)

Beschreibung

Summary

Define the lifecycle for signal models and extractors, including datasets, evaluation, versioning, release provenance, and runtime compatibility.

Parent roadmap: #2287 Parent signal roadmap: #2335 Related: #1485, #2154, #2247, #2301, #2347, #2382 Related code: src/training, src/semantic-router/pkg/classification, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/modelinventory, candle-binding, nlp-binding, ml-binding

Motivation

H2 includes improving signal model accuracy and adding more extractors. The repo already has training/eval directories and runtime model bindings, but roadmap issues mostly describe individual models or bugs. Contributors need a lifecycle contract: what dataset/eval evidence is required, how model versions are published, how confidence calibration is measured, and how runtime compatibility is validated.

Scope

Define lifecycle requirements for:

  • training and evaluation datasets;
  • domain, complexity, modality, rerank, PII, feedback, hallucination, and custom signal extractors;
  • confidence calibration and threshold provenance;
  • model artifact versioning and catalog entries;
  • CPU/Candle/ONNX/native binding compatibility;
  • release notes and rollback behavior for signal model changes;
  • base-model and model-family research, publication, and migration requirements for next-generation router reranker, embedding, and classifier models (#2382).

Non-goals

  • Do not require every signal to use a learned model.
  • Do not accept accuracy improvements without reproducible eval evidence.
  • Do not silently change default thresholds without provenance.

Acceptance criteria

  • Signal model changes have dataset/eval/provenance requirements.
  • Runtime model catalogs can identify model version, capability, and compatibility constraints.
  • Accuracy and calibration reports are linked from relevant signal issues.
  • Default signal model updates include rollback guidance.

Contributor Guide