vllm-project/semantic-router

feat: define signal model lifecycle, datasets, and release provenance

開放

#2,360 建立於 2026年7月5日

 (0 則留言) (0 個反應) (0 位負責人)Go (699 個分叉)github user discovery
area/researchenhancementenvironment/rocmevaluationhelp wantedpriority/P1roadmapsignal-decision-engine

倉庫指標

星標
 (4,293 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Summary

Define the lifecycle for signal models and extractors, including datasets, evaluation, versioning, release provenance, and runtime compatibility.

Parent roadmap: #2287 Parent signal roadmap: #2335 Related: #1485, #2154, #2247, #2301, #2347, #2382 Related code: src/training, src/semantic-router/pkg/classification, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/modelinventory, candle-binding, nlp-binding, ml-binding

Motivation

H2 includes improving signal model accuracy and adding more extractors. The repo already has training/eval directories and runtime model bindings, but roadmap issues mostly describe individual models or bugs. Contributors need a lifecycle contract: what dataset/eval evidence is required, how model versions are published, how confidence calibration is measured, and how runtime compatibility is validated.

Scope

Define lifecycle requirements for:

  • training and evaluation datasets;
  • domain, complexity, modality, rerank, PII, feedback, hallucination, and custom signal extractors;
  • confidence calibration and threshold provenance;
  • model artifact versioning and catalog entries;
  • CPU/Candle/ONNX/native binding compatibility;
  • release notes and rollback behavior for signal model changes;
  • base-model and model-family research, publication, and migration requirements for next-generation router reranker, embedding, and classifier models (#2382).

Non-goals

  • Do not require every signal to use a learned model.
  • Do not accept accuracy improvements without reproducible eval evidence.
  • Do not silently change default thresholds without provenance.

Acceptance criteria

  • Signal model changes have dataset/eval/provenance requirements.
  • Runtime model catalogs can identify model version, capability, and compatibility constraints.
  • Accuracy and calibration reports are linked from relevant signal issues.
  • Default signal model updates include rollback guidance.

貢獻者指南