feature: make Fusion adaptive and support repeated self-sampling
#2,865 opened on 2026/08/11
Repository metrics
- Stars
- (4,293 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
Summary
Reduce Fusion's fixed N+2 cost and support repeated sampling from the same model while preserving Fusion's panel-analysis-synthesis semantics.
Parent hardening tracker: #2336
Blocked by: #1456, #2855, #2856, #2857, #2858, #2861 Related grounding evidence: #2860
Current behavior and gaps
Fusion executes a parallel panel, optional grounding, one structured judge-analysis call, and one final synthesis call. It currently:
- normalizes/deduplicates analysis model names, so repeating one model does not create true independent self-fusion samples;
- always pays the analysis call even when
include_analysis=falseonly suppresses trace output; - runs judge analysis and final synthesis sequentially using the same judge model;
- pays the full panel plus judge stages for easy requests;
- has only limited evidence that panel-consistency grounding helps; hard filtering is known to hurt contested factual QA and soft weighting remains under evaluation in #2860.
Proposed phases
Repeated sampling
- Add explicit
samples_per_modelor candidate-call specifications independent of unique model identity. - Apply per-sample temperature/seed/reasoning controls and preserve repeated attempts in trace/usage.
- Report same-model and heterogeneous-panel operation distinctly.
Judge cost reduction
- Skip the analysis model call when analysis is genuinely disabled.
- Evaluate a single structured judge response containing bounded analysis plus final answer, compared against the current two-call pipeline.
- Keep a fallback compatible with current raw-panel synthesis when structured parsing fails.
Adaptive gate
- Use Router evidence, a cheap lead/probe answer, or verifier confidence to choose direct return, reduced panel, or full Fusion.
- Reserve every stage from #1456/#2861 before dispatch.
- Emit the gate decision and counterfactual budget class in Replay.
Research context
ICLR 2026 Fusion-of-N reports benefits from synthesizing complementary samples rather than selecting one. NeurIPS 2025 evidence on overthinking favors parallel samples over extending one chain. These motivate repeated self-sampling, but not unconditional full-panel spend.
Non-goals
- Do not change grounding defaults in this issue; #2860 decides
off|annotate|weightefficacy. - Do not turn Fusion into Best-of-N selection.
- Do not add step-level tree search or arbitrary tool trajectories.
Acceptance criteria
- Repeated same-model candidates are represented as distinct calls with complete accounting.
-
include_analysis=falseno longer pays an unused analysis call, with corrected iterations/usage. - A one-call structured judge mode is evaluated against the two-call baseline for parse reliability, quality, tokens, and latency.
- Adaptive gating has deterministic direct/reduced/full outcomes and cannot exceed hard budgets.
- Cached-panel tests isolate judge changes from panel sampling noise.
- Fixed-budget benchmarks report same-model and heterogeneous panels separately.
- E2E covers repeated samples, skipped analysis, gate outcomes, partial panels, and fallback.