vllm-project/semantic-router

bench: complete cached-panel Grounding-aware Fusion efficacy A/B

Ouverte

#2 860 ouverte le 11 août 2026

 (1 commentaire) (0 réaction) (0 personne assignée)Go (699 forks)github user discovery
area/bencharea/momarea/researchenhancementevaluationhelp wantedpriority/P1roadmap

Métriques du dépôt

Stars
 (4 293 étoiles)
Métriques de merge PR
 (Métriques PR en attente)

Description

Summary

Complete the paired cached-panel evaluation needed to decide whether Grounding-aware Fusion weight or annotate improves final synthesis over plain Fusion.

Parent: #2336 Related algorithm research: #2337

Current evidence

The existing DRACO evaluation established two important facts:

  • panel-NLI scores became discriminative after scorer fixes, but only weakly correlated with response quality;
  • hard filter significantly regressed contested factual questions by deleting correct minority/dissenter evidence.

The production default therefore moved to soft weight, but the repository still has no completed paired evidence that weight or annotate beats off rather than merely avoiding the known hard-filter harm.

Scope

Using byte-identical cached panels, compare:

  • off: plain Fusion;
  • annotate: expose scores as notes without weighting instructions;
  • weight: soft weighting with dissenter protection;
  • deterministic placebo/random scores to test whether any gain comes from the signal rather than prompt shape.

Keep filter only as a known-risk reference arm where useful. Run Level-1 scorer analysis and Level-2 final-answer analysis with paired bootstrap confidence intervals, including contested slices and domain breakdowns.

Non-goals

  • Do not re-enable hard filtering by default.
  • Do not claim peer consistency is factual truth.
  • Do not mix panel regeneration noise into selector-policy comparisons.
  • Do not add context-mode claims without a dataset that provides trusted source documents.

Acceptance criteria

  • Every arm synthesizes from the same cached panel for each item.
  • Reports include scorer correlation, final quality delta, negative-criteria delta, and paired confidence intervals.
  • Contested items and correct-minority/dissenter cases are reported separately.
  • Placebo scores distinguish real-signal value from prompt/weighting effects.
  • A predeclared keep/disable decision is recorded for weight and annotate.
  • filter remains opt-in and documented as known-risk unless new evidence overturns the current finding.
  • Follow-up context-mode evaluation is split into a separate RAG-grounded issue if warranted.

Validation

  • Deterministic cached-panel fixture tests.
  • Smoke run before the larger Medicine/Law or factuality slice.
  • Machine-readable per-item and summary artifacts with reproducible commands.

Guide contributeur