vllm-project/semantic-router

feat: define signal model lifecycle, datasets, and release provenance

Aperta

#2360 aperta il 5 lug 2026

 (0 commenti) (0 reazioni) (0 assegnatari)Go (699 fork)github user discovery
area/researchenhancementenvironment/rocmevaluationhelp wantedpriority/P1roadmapsignal-decision-engine

Metriche repository

Star
 (4293 stelle)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Summary

Define the lifecycle for signal models and extractors, including datasets, evaluation, versioning, release provenance, and runtime compatibility.

Parent roadmap: #2287 Parent signal roadmap: #2335 Related: #1485, #2154, #2247, #2301, #2347, #2382 Related code: src/training, src/semantic-router/pkg/classification, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/modelinventory, candle-binding, nlp-binding, ml-binding

Motivation

H2 includes improving signal model accuracy and adding more extractors. The repo already has training/eval directories and runtime model bindings, but roadmap issues mostly describe individual models or bugs. Contributors need a lifecycle contract: what dataset/eval evidence is required, how model versions are published, how confidence calibration is measured, and how runtime compatibility is validated.

Scope

Define lifecycle requirements for:

  • training and evaluation datasets;
  • domain, complexity, modality, rerank, PII, feedback, hallucination, and custom signal extractors;
  • confidence calibration and threshold provenance;
  • model artifact versioning and catalog entries;
  • CPU/Candle/ONNX/native binding compatibility;
  • release notes and rollback behavior for signal model changes;
  • base-model and model-family research, publication, and migration requirements for next-generation router reranker, embedding, and classifier models (#2382).

Non-goals

  • Do not require every signal to use a learned model.
  • Do not accept accuracy improvements without reproducible eval evidence.
  • Do not silently change default thresholds without provenance.

Acceptance criteria

  • Signal model changes have dataset/eval/provenance requirements.
  • Runtime model catalogs can identify model version, capability, and compatibility constraints.
  • Accuracy and calibration reports are linked from relevant signal issues.
  • Default signal model updates include rollback guidance.

Guida contributor