vllm-project/semantic-router

feat: define signal model lifecycle, datasets, and release provenance

开放

#2,360 创建于 2026年7月5日

 (0 条评论) (0 个反应) (0 位负责人)Go (699 个派生)github user discovery
area/researchenhancementenvironment/rocmevaluationhelp wantedpriority/P1roadmapsignal-decision-engine

仓库指标

星标
 (4,293 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Summary

Define the lifecycle for signal models and extractors, including datasets, evaluation, versioning, release provenance, and runtime compatibility.

Parent roadmap: #2287 Parent signal roadmap: #2335 Related: #1485, #2154, #2247, #2301, #2347, #2382 Related code: src/training, src/semantic-router/pkg/classification, src/semantic-router/pkg/modelruntime, src/semantic-router/pkg/modelinventory, candle-binding, nlp-binding, ml-binding

Motivation

H2 includes improving signal model accuracy and adding more extractors. The repo already has training/eval directories and runtime model bindings, but roadmap issues mostly describe individual models or bugs. Contributors need a lifecycle contract: what dataset/eval evidence is required, how model versions are published, how confidence calibration is measured, and how runtime compatibility is validated.

Scope

Define lifecycle requirements for:

  • training and evaluation datasets;
  • domain, complexity, modality, rerank, PII, feedback, hallucination, and custom signal extractors;
  • confidence calibration and threshold provenance;
  • model artifact versioning and catalog entries;
  • CPU/Candle/ONNX/native binding compatibility;
  • release notes and rollback behavior for signal model changes;
  • base-model and model-family research, publication, and migration requirements for next-generation router reranker, embedding, and classifier models (#2382).

Non-goals

  • Do not require every signal to use a learned model.
  • Do not accept accuracy improvements without reproducible eval evidence.
  • Do not silently change default thresholds without provenance.

Acceptance criteria

  • Signal model changes have dataset/eval/provenance requirements.
  • Runtime model catalogs can identify model version, capability, and compatibility constraints.
  • Accuracy and calibration reports are linked from relevant signal issues.
  • Default signal model updates include rollback guidance.

贡献者指南