bench: complete cached-panel Grounding-aware Fusion efficacy A/B
#2.860 aberto em 11 de ago. de 2026
Métricas do repositório
- Stars
- (4.293 estrelas)
- Métricas de merge de PR
- (Métricas PR pendentes)
Description
Summary
Complete the paired cached-panel evaluation needed to decide whether Grounding-aware Fusion weight or annotate improves final synthesis over plain Fusion.
Parent: #2336 Related algorithm research: #2337
Current evidence
The existing DRACO evaluation established two important facts:
- panel-NLI scores became discriminative after scorer fixes, but only weakly correlated with response quality;
- hard
filtersignificantly regressed contested factual questions by deleting correct minority/dissenter evidence.
The production default therefore moved to soft weight, but the repository still has no completed paired evidence that weight or annotate beats off rather than merely avoiding the known hard-filter harm.
Scope
Using byte-identical cached panels, compare:
off: plain Fusion;annotate: expose scores as notes without weighting instructions;weight: soft weighting with dissenter protection;- deterministic placebo/random scores to test whether any gain comes from the signal rather than prompt shape.
Keep filter only as a known-risk reference arm where useful. Run Level-1 scorer analysis and Level-2 final-answer analysis with paired bootstrap confidence intervals, including contested slices and domain breakdowns.
Non-goals
- Do not re-enable hard filtering by default.
- Do not claim peer consistency is factual truth.
- Do not mix panel regeneration noise into selector-policy comparisons.
- Do not add context-mode claims without a dataset that provides trusted source documents.
Acceptance criteria
- Every arm synthesizes from the same cached panel for each item.
- Reports include scorer correlation, final quality delta, negative-criteria delta, and paired confidence intervals.
- Contested items and correct-minority/dissenter cases are reported separately.
- Placebo scores distinguish real-signal value from prompt/weighting effects.
- A predeclared keep/disable decision is recorded for
weightandannotate. -
filterremains opt-in and documented as known-risk unless new evidence overturns the current finding. - Follow-up context-mode evaluation is split into a separate RAG-grounded issue if warranted.
Validation
- Deterministic cached-panel fixture tests.
- Smoke run before the larger Medicine/Law or factuality slice.
- Machine-readable per-item and summary artifacts with reproducible commands.