feat: define signal model lifecycle, datasets, and release provenance
#2,360 opened on Jul 5, 2026
Repository metrics
- Stars
- (4,293 stars)
- PR merge metrics
- (PR metrics pending)
Description
Summary
Define the lifecycle for signal models and extractors, including datasets, evaluation, versioning, release provenance, and runtime compatibility.
Parent roadmap: #2287
Parent signal roadmap: #2335
Related: #1485, #2154, #2247, #2301, #2347, #2382
Related code: src/training, src/semantic-router/pkg/classification, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/modelinventory, candle-binding, nlp-binding, ml-binding
Motivation
H2 includes improving signal model accuracy and adding more extractors. The repo already has training/eval directories and runtime model bindings, but roadmap issues mostly describe individual models or bugs. Contributors need a lifecycle contract: what dataset/eval evidence is required, how model versions are published, how confidence calibration is measured, and how runtime compatibility is validated.
Scope
Define lifecycle requirements for:
- training and evaluation datasets;
- domain, complexity, modality, rerank, PII, feedback, hallucination, and custom signal extractors;
- confidence calibration and threshold provenance;
- model artifact versioning and catalog entries;
- CPU/Candle/ONNX/native binding compatibility;
- release notes and rollback behavior for signal model changes;
- base-model and model-family research, publication, and migration requirements for next-generation router reranker, embedding, and classifier models (#2382).
Non-goals
- Do not require every signal to use a learned model.
- Do not accept accuracy improvements without reproducible eval evidence.
- Do not silently change default thresholds without provenance.
Acceptance criteria
- Signal model changes have dataset/eval/provenance requirements.
- Runtime model catalogs can identify model version, capability, and compatibility constraints.
- Accuracy and calibration reports are linked from relevant signal issues.
- Default signal model updates include rollback guidance.