feat: add unified signal registry and observation protocol

Summary

Add a unified signal registry and observation protocol for both built-in signals and user-defined signal extensions. The goal is to preserve the existing signals -> projections -> decisions -> strategy/model_selection pipeline while removing the need to add new hard-coded result fields and dispatch paths for every new signal family.

Motivation

The current signal runtime is family-specific across config, dispatch, result storage, projections, and decision evaluation. Built-in families such as keyword, domain, embedding, PII, language, and complexity are represented as fixed config/result fields. That makes new signal types expensive to add and prevents users from extending signal extraction without changing router code.

Proposed design

Introduce a global signal registry. Built-in signals are registered implicitly by the router and do not need to be declared by users. User-defined signals are declared under global.signals.

global:
  signals:
    - name: tenant_context
      protocol: http
      endpoint: http://tenant-service/signals
      schema: vllm.ai.semantic-router.signal.v1
      timeout_ms: 50
      outputs:
        - name: tenant_risk
          type: score
          range: [0, 1]
        - name: tenant_tier
          type: label

Runtime built-ins should normalize into the same registry model, for example:

global:
  signals:
    - name: keyword
      protocol: builtin
      schema: vllm.ai.semantic-router.signal.v1
    - name: domain
      protocol: builtin
      schema: vllm.ai.semantic-router.signal.v1
    - name: embedding
      protocol: builtin
      schema: vllm.ai.semantic-router.signal.v1

Recipes or legacy routing config should be able to enable an extension signal by using kind as the registry key and name as the recipe-local signal name referenced by projections and decisions:

signals:
  custom:
    - name: privacy_risk
      kind: tenant_context
      outputs: ["tenant_risk"]
      required: true

No separate signalProvider abstraction is needed. protocol describes how the signal is executed (builtin, http, grpc, later wasm if needed). schema describes the wire contract.

Unified observation contract

All signal extractors should produce normalized observations, including built-ins:

{
  "schema": "vllm.ai.semantic-router.signal.v1",
  "observations": [
    {
      "signal": "privacy_risk",
      "output": "tenant_risk",
      "status": "ok",
      "matched": true,
      "confidence": 0.91,
      "value": 0.84,
      "label": "high",
      "metadata": {}
    }
  ]
}

The status model should distinguish successful matches from runtime failures, for example ok, no_match, timeout, unavailable, invalid_input, and error, so routing can apply explicit fail-open/fail-closed behavior.

Affected areas

• src/semantic-router/pkg/config/**
• src/semantic-router/pkg/classification/**
• src/semantic-router/pkg/decision/**
• src/semantic-router/pkg/extproc/req_filter_classification.go
• config docs and examples under config/, deploy/recipes/, and website/docs/

Acceptance criteria

• Built-in signals are registered implicitly in the same runtime registry as user-defined signals.
• Existing built-in signal config remains compatible.
• global.signals can register an HTTP-based custom signal using schema: vllm.ai.semantic-router.signal.v1.
• Custom signals can be enabled in routing through signals.custom[].kind and given recipe-local names through signals.custom[].name.
• Built-in and custom extractors both produce a common SignalObservation-style result.
• Projections and decisions can reference custom signal observations without adding a new hard-coded Go field per signal family.
• Diagnostics or normalized config output expose implicit built-ins and configured custom signals for debugging.

Validation

Run the repo harness for the touched config/classification/decision surfaces:

make agent-report ENV=cpu CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/classification src/semantic-router/pkg/decision src/semantic-router/pkg/extproc config deploy/recipes website/docs"
make agent-lint CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/classification src/semantic-router/pkg/decision src/semantic-router/pkg/extproc config deploy/recipes website/docs"
make agent-ci-gate CHANGED_FILES="src/semantic-router/pkg/config src/semantic-router/pkg/classification src/semantic-router/pkg/decision src/semantic-router/pkg/extproc config deploy/recipes website/docs"