Apple's on-device model (fm) for TokenEconomizer's side calls and audio prompts: approach, rationale and results

Start with PR #185 and the documented sidecall.runner_for flow, then review the fm_call and SpeechAnalyzer/SpeechTranscriber implementation and the evidence bundle mentioned in the issue. The issue describes opt-in on-device judging and audio transcription, their fallback limits, and measured results; the work involves evaluating and integrating behavior across model calls, prompt hooks, and macOS-specific tooling.

PR: #185

macOS 27 ships Apple's Foundation Models CLI at /usr/bin/fm. It runs a ~3B model ("AFM 3 Core") on the Neural Engine and GPU, with a 4,096-token window, guided JSON-schema output, image input, and no login or key. That made me ask where an on-device model helps TokenEconomizer, and where it doesn't. This issue covers the approach, the rationale, and every experiment, including the negative ones. The PR adds the two options that held up, both opt-in.

Where an on-device model could plug in

All three of TokenEconomizer's model calls go through sidecall.runner_for(host, config) → runner(prompt, model) -> (text, effective_tokens):

Side call	What it asks	Fit for a 3B model with a 4k window	Result
Stuck judge	a ≤1,500-token digest → {stuck, confidence, reason}	a classification task, which suits guided output	✅ usable with a schema and a calibrated floor (below)
Redirect writer	the digest → 2–4 sentences	short constrained text	⚠️ follows every hard rule, but generic
Fact proposer	8 session digests → facts with Python-regex detectors and citations	long context plus authoring code	❌ no usable facts (below)

On Claude Code each side call is a claude -p call that carries the CLI's own base prompt. Your docs measured about 14k effective tokens for a real judge call, and about 25k for two verdicts plus a redirect. I measured 6,981 effective tokens per judge call on Haiku 5.5.

1. The judge

Data. I used 502 SWE-agent trajectories from the public nebius/SWE-agent-trajectories, 94 of them resolved. SWE-agent commands map onto TE's tool names: open/goto/scroll → Read, edit → Edit, create → Write, search_*/find_file → Grep, the rest → Bash. I replayed each trajectory call by call through loopguard.TurnGuard with default thresholds. The envelope is p90 of resolved lengths × 1.25 = 40 calls, shared because there are too few resolved runs per repo. A recording judge that always spares collected every digest the guard would have sent, up to 3 per trajectory: 550 points. I judged a stratified sample of 302 with llm_stuck_judge itself:

• 52 points from resolved runs, where a redirect interrupts a success.
• 150 from runs that ran out of their context budget, the likeliest loops.
• 100 from other unresolved runs.

Haiku went through TE's own claude_call; fm went through the PR's fm_call.

Judge	threshold	resolved ↓	exhausted	other	AUC (exhausted vs resolved)	tokens/call	s/call
Haiku 5.5, current default	0.70	26/52 (50%)	115/150 (77%)	53/100 (53%)	0.684	6,981	3.9
Haiku 5.5	0.85	10/52 (19%)	63/150 (42%)	21/100 (21%)	0.684	6,981	3.9
fm, verdict-first schema (the PR)	0.70	37/52 (71%)	123/150 (82%)	76/100 (76%)	0.639	0	2.0
fm, verdict-first schema (the PR)	0.90	7/52 (13%)	56/150 (37%)	13/100 (13%)	0.639	0	2.0
fm, evidence-first schema	0.70	48/52 (92%)	143/150 (95%)	89/100 (89%)	0.638	0	4.5
fm, evidence-first schema	0.90	11/52 (21%)	71/150 (47%)	19/100 (19%)	0.638	0	4.5
fm, free text (no schema)	0.70	4/52 (8%)	13/150 (9%)	2/100 (2%)	0.383	0	1.9

What I take from this:

• A schema is required. In free text fm rarely says stuck, and it doesn't separate loops from successes: 9% of exhausted points against 8% of resolved ones, AUC 0.38. In 182 of its 283 "not stuck" verdicts the reason contains a loop phrase such as "repeated identical failures" (a keyword count; it also catches some hedged reasons). The schema helps but doesn't remove this: verdict-first still has 38 of 66, evidence-first 5 of 16, Haiku 7 of 104 (mostly hedged, such as "repeated the same failures, but then shifted to reading the codebase"). fm respond --schema constrains decoding, so the verdict always parses.
• A floor is required. fm's confidences are quantized (0.65/0.72/0.85/0.92). At the guard's 0.7 nearly every "stuck" counts, so the PR adds local_judge_confidence=0.9, applied only when fm answered.
• At 0.9, fm is close to Haiku's own trade-off curve but slightly below it (AUC 0.64 vs 0.68). It redirects fewer resolved runs than Haiku at 0.85 (13% vs 19%) and catches fewer exhausted runs (37% vs 42%). I chose the verdict-first schema because it is the more conservative of the two ("protect task success") and twice as fast. Evidence-first, which writes progress and repetition signs before the verdict, catches more loops (47%) and contradicts itself less, at 21% on resolved runs. It is a one-line swap if you prefer that point.
• Haiku at the default 0.7 redirected half of the resolved runs' decision points too. With the benchmark-default thresholds the judge does a lot of the work. That is about the guard's settings, not about fm.
• Cost: the 302 Haiku calls were 2.1M effective tokens. On fm they are 0, and the call returns in about 2 s.

Live, Claude Code 2.1.319 with the branch (--plugin-dir, TE_LOCAL_MODEL=fm). I ran 16 Haiku sessions in a toy repo: solvable tasks, plus tasks whose tests contradict each other. The judge ran on fm ("local": "fm", _tokens: 0 in its verdicts), and nothing was charged to the ledger. Haiku 5.5 does not loop on its own: it notices contradictory tests within 3–6 turns and stops, even when told to keep trying. A scripted 8-cycle loop drew one early verdict, a spare at 0.65 after about 6 calls. The backoff then moved the next check past the turn's end. Re-judged on the turn's final digest, fm says stuck at 0.92.

2. The redirect writer (local_roles=judge,nudge, opt-in)

On stuck digests it never told the agent to stop, never named line numbers, and never invented identifiers. But it mostly repeats the prompt's own examples ("re-read the failing test's expectation") and gets details wrong: it says "same file" for two files, and talks about a "failing test" for a make loop. It is probably no better than rules.nudge_text, so it stays opt-in and the default role list is judge.

3. The proposer: not a fit

I ran TE's Distiller over 15 iterative/dvc trajectories converted to Claude Code transcripts. The prompt was 8,856 characters (2,780 fm tokens), so it fits.

• Haiku: one call, 42.5k effective tokens, 32 s. It proposed two facts consistent with the sessions (the package layout, and "scripts run against the working tree").
• fm, free text: invalid JSON. Its regex escaping broke.
• fm with a schema: valid JSON and valid regexes, but invented facts ("dvc build", "dvc lint", "pip install dvc"), with detectors that would never match.

local_roles=proposer exists for completeness, but I would not recommend it.

4. Audio prompts: a clear win (media=audio)

Claude Code's Read rejects an audio file as binary before PreToolUse runs, so a Read hook cannot help. I verified that: the hook never fired. The prompt hook can. The task, with a Haiku 5.5 agent, 5 runs per arm and hidden tests: "the PM left a voice memo notes/bug_memo.aiff; implement exactly what it asks" (round half up, reject negative or >1 percentages).

Arm	pass	cost	effective tokens	turns
no plugin	0/5	$0.0060	54,269	12.8
Read hook (never fires for audio)	0/5	$0.0051	45,983	11.6
standalone prompt hook	5/5	$0.0041	35,483	7.6
this branch, TE_MEDIA=audio	5/5	$0.0035 (−42%)	30,306 (−44%)	7.6

The transcript contains recognition errors ("Refunds Dog Pie" for refunds.py, "sand" for "cent"). The agent read through them every time. The transcriber is Apple's SpeechAnalyzer/SpeechTranscriber: 38× realtime on an M2 in my tests, on-device, with no entitlement needed. It is a 40-line Swift helper compiled once with swiftc.

5. Images: no win, so not in the PR

I tried a PreToolUse(Read) hook that hands the agent fm's text transcription of a screenshot (updatedInput → a markdown file; a second Read of the same path gets the pixels). fm transcribed the test screenshots almost exactly. But cost didn't move, and every run passed:

Task (5 runs per arm)	base cost / eff tokens	transcription cost / eff tokens
debug from a traceback screenshot	$0.0039 / 33,817	$0.0037 / 32,648
match a pricing-table screenshot	$0.0032 / 26,593	$0.0032 / 27,709
four QA screenshots	$0.0099 / 83,865	$0.0096 / 85,235

A screenshot is a small share of a session, and the transcription is comparable in size. The hook also adds 3–6 s per image. So no image mechanism.

Limits

• Only on macOS 27 with Apple Intelligence on, Apple silicon. On other systems both options do nothing and fall back.
• fm's window is 4,096 tokens. The judge's and redirect writer's prompts fit with room to spare: the largest of the 550 replayed digests, with the judge's instructions, is 1,359 fm tokens.
• The judge numbers are an offline replay on another agent's trajectories, not a rerun of the paper's arms, which need the Linux/Docker harness. A rerun of spiral-only with local_model=fm against Haiku on a few repos would be the decisive test. I'm glad to help with it on a Mac host if the harness can reach one.

Evidence

The scripts and raw per-point results are in an evidence bundle:

• The replay through TurnGuard.
• The judge harness that calls llm_stuck_judge.
• All 302 × 4 verdicts (Haiku; fm verdict-first, evidence-first and free text).
• The media A/B runner with its hidden tests.
• The proposer runs.
• The live session log.

I'll post a link to the bundle in a comment on this issue.