bench: compare Looper TTS algorithms under fixed quality-cost budgets

Summary

Add a fixed-budget quality/cost benchmark matrix for Looper test-time scaling algorithms using frozen prompts, model pools, and scoring protocols.

Parent: #2336 Extends the completed component/regression baseline: #2371 Related algorithm research: #2337

Motivation

The repository has strong end-to-end Router Flow results and component performance baselines, but current scores use different recipes, model pools, budgets, verifiers, and finalizers. They cannot isolate whether quality gains come from the algorithm, a stronger model, more tokens, or a different judge.

Before adding adaptive allocation or new algorithms, the project needs comparable quality-versus-cost curves.

Scope

Compare at least:

• direct single-model generation;
• confidence cascade;
• ReMoM breadth schedules;
• Fusion panel/judge/synthesis.

Use frozen model pools and prompt sets. Reuse byte-identical cached candidate panels when comparing selection or synthesis policies.

Record:

• pass@1 and final selected accuracy/score;
• oracle pass@N and verifier/selector regret where applicable;
• attempted/successful calls;
• prompt, completion, and total tokens;
• estimated provider cost from model pricing;
• wall latency and p50/p95 distributions;
• paired confidence intervals and easy/hard/contested slices.

Start with bounded GPQA-Diamond and HLE slices, then add LiveCodeBench and RAG-grounded factuality when adapters are stable.

Non-goals

• Do not replace benchmark-native or EvalScope scoring.
• Do not publish synthetic smoke scores as full benchmark claims.
• Do not compare algorithms with undisclosed model-pool upgrades.
• Do not add a new production algorithm in this issue.

Acceptance criteria

• Direct, confidence, ReMoM, and Fusion can be compared under at least two matched budgets.
• Every result records the exact config, model pool, prompt set, budget, scorer, raw output location, and command.
• Cached-panel comparisons isolate selector/fuser behavior from generation noise.
• Reports include quality, tokens, estimated cost, p50/p95 latency, and paired uncertainty.
• Same-model and mixed-model results are reported separately.
• The harness emits machine-readable artifacts suitable for future adaptive-N gates.

Validation

• Unit tests for result normalization, cost computation, pairing, and report generation.
• A small deterministic fixture in CI.
• One reproducible live smoke before any larger campaign.