Benchmark: run the paired premise-checking A/B on Claude (Copilot SDK and Claude Agent SDK)
Les mainteneurs répondent en général sous 1 jour
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Accessibilité débutants
- 25/100
- Type d'issue
- Fonctionnalité
- Clarté
- Plutôt claire
- Activité
- Active
- Stack technique
- github, javascript, typescript
- Domaine
- ai-infra-agents, testing
Piste de recherche
Start with scripts/premise-checking-ab.mjs on the bradygaster-premise-checking-plan branch and review the existing Copilot SDK calls and benchmark protocol. Then verify the Claude Agent SDK mapping against its current docs before extracting HostDriver and adding the Claude driver. Done means a frozen paired row matches its protocol hashes under the unchanged Copilot driver and the Claude host completes a pilot and paired calibration with the required usage and ledger data.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Summary
Run Squad's paired premise-checking A/B benchmark (scripts/premise-checking-ab.mjs, on branch bradygaster-premise-checking-plan) on Claude. That gives Claude users a measured "Claude alone vs. Claude + Squad" comparison. Related: #2438 (Claude Code projection).
Claude users will ask two separate questions:
- Model: does Squad help when the model is Claude?
- Harness: does Squad help inside Claude Code itself?
Results from Copilot alone answer neither.
Design: one fixture, varying only the host
| Copilot SDK (existing driver) | Claude Agent SDK (new driver) | |
|---|---|---|
| Claude model | Phase A (done for calibration, below) | Phase B |
| gpt-5.3-codex | existing results | n/a |
The fixture, oracle criteria, arms, prompt sources, budgets, blinding and adjudication all stay frozen. Pin one Claude model ID across both phases.
Phase A: Claude model through the Copilot SDK
Implemented on branch squad/claude-ab-pilot (commit fae48038):
--model <id>: recorded in the run manifest. Resuming a campaign under a different model is refused. The default staysgpt-5.3-codex, so existing manifests still validate.--reviewer-check: a scripted truth-first run that is told to dispatch the Fact Checker once. It verifies reviewer accounting end to end. It is not a measured arm.--revalidate-pilot: accepts a pilot with no Fact Checker dispatch only if a reviewer check passed on the same model.- Dispatch counts: pilot and paired validation now report Fact Checker dispatches per arm instead of failing when there are none.
Phase B: Claude Agent SDK driver (to do)
- Driver interface: put the harness's Copilot SDK calls behind a
HostDriverinterface (preflight,createSession,sendAndWait,abort, normalizedmessage/usage/toolevents). Re-run one frozen paired row and match its protocol hashes to prove the Copilot driver is unchanged. - Claude driver: add
--host claudeusingquery().- Custom tools go through an in-process MCP server.
allowedToolsis limited toread_evidenceandtask, andsettingSources: []keeps user and project config out.canUseToolrejects everything else.- Usage comes from assistant messages plus
total_cost_usdon the result.
- Fact Checker dispatch: keep it as a separate
query()session, as it is today. NativeAgent-tool dispatch can come later as an optional fourth arm (truth-first-native). - Ledger: record cache-read and cache-write input separately, but charge budgets on all input classes combined.
- Verify the mapping: check it against current Claude Agent SDK docs before building.
Results so far (2026-11-09, claude-sonnet-5.5 through Copilot)
Pilot (revenue-midflight-agent, all 3 arms):
- 2 coordinator calls per episode.
- About 78K input tokens per episode, roughly half of them cache reads, and about 310 output tokens.
- 7–20 s per episode.
- Claude made no Fact Checker dispatch. That failed the original gate until the reviewer check passed (2 reviewer calls, about 17K input tokens, every evidence read traced).
Paired calibration (16 runs: 8 families × baseline/truth-first):
- 16 of 16 completed, and the instrumentation check passed.
- 927,615 input tokens (340K cache reads, 587K cache writes) and 5,743 output tokens, in 125 s altogether.
- About $1.60 at API rates on Sonnet 5.5 and about $3.10 on Opus 5.5.
- Fact Checker dispatches: 0 of 8 in baseline and 0 of 8 in truth-first.
- Before blinded adjudication:
- Both arms refused every unsupported premise and kept authorized work going.
- Truth-first answers state more often what wasn't checked or dispatched.
- Blinded human adjudication is still to do.
Takeaway: on this fixture, Claude handles the premises carefully at baseline without a reviewer call. A Claude write-up shouldn't promise a large truth-first gain here. Instead it should:
- report the reviewer cost Claude avoids;
- add harder episodes, with ambiguous evidence or claims that would trigger an action, where review should matter.
Projected cost (Claude API rates, plus about 50% for retries)
| Campaign | Sonnet 5.5 | Opus 5.5 |
|---|---|---|
| Paired calibration (16) | ~$2–4 | ~$4–8 |
| Full plan (20 episodes × 3 arms) | ~$8–15 | ~$16–30 |
Through Copilot, runs use premium requests instead. Phase B adds Claude Code's own system prompt when using the claude_code preset, so measure that in its pilot.
Scope
This is a prompt-behavior evaluation on a synthetic fixture, not a coding benchmark. Pair it with a separate coding benchmark, for example a SWE-bench Verified subset comparing Claude Code alone with Claude Code + Squad.
Tasks
- Land
squad/claude-ab-pilot(or rebase it onto the harness branch once that is ondev) - Blinded adjudication of the Sonnet 5.5 paired calibration
- Harder fixture episodes where review should change the outcome
- Extract
HostDriver; prove the Copilot driver unchanged - Claude Agent SDK driver (
--host claude): pilot, then paired calibration - Optional
truth-first-nativearm - Publish the write-up: per-family tables, tokens/cost/time, manifests, raw JSONL, and a one-command rerun
- Langage dominant
- TypeScript
- Étoiles
- 3.4k
- Forks
- 516
- Merge moyen
- 1 j 15 h
- PR mergées (30 j)
- 116
Préparer son environnement
- Fournit un Dockerfile ou un fichier Docker Compose
- Propose un modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de github/squad
-
bug
Difficulté 2/5 1-3 heures Accessibilité débutants 84/100
Les mainteneurs répondent en général sous 1 jour
-
bug
Difficulté 2/5 1-3 heures Accessibilité débutants 77/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de github/squad
Issues similaires
-
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
vercel-labs/ui#809 ·
Les mainteneurs répondent en général sous 1 jour
-
bot:ai-assisted component:platform-js status:untriaged
Difficulté 1/5 Moins d'une heure Accessibilité débutants 74/100
midnightntwrk/platform-js#38 ·
-
[bug] Second false positive on high-cost plugin alert: detection only compares patch entry id, community plugin dsh-auto | [bug] 高成本插件告警第二次误报:检测只比对 patch 条目 id,社区插件 dsh-auto-review 被认成 @deepseek-ai/dsh-experimental-auto-reviewPeut-être pris Une pull request liée à cette issue est ouverte ou déjà fusionnée. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
ranxianglei/billion-context#7068 ·
Les mainteneurs répondent en général sous 1 jour
-
vendor: bump whisper.cpp past 6a8addd — the current pin aborts the app on a failed Metal initOuverte
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
mybigday/whisper.rn#348 ·
-
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
six-group/six-webcomponents#600 ·