Apple's on-device model (fm) for TokenEconomizer's side calls and audio prompts: approach, rationale and results

Start with PR #185 and the documented sidecall.runner_for flow, then review the fm_call and SpeechAnalyzer/SpeechTranscriber implementation and the evidence bundle mentioned in the issue. The issue describes opt-in on-device judging and audio transcription, their fallback limits, and measured results; the work involves evaluating and integrating behavior across model calls, prompt hooks, and macOS-specific tooling.

PR: #185

Update after review: #185 now adds only the on-device judge (local_model=fm), with the review's fixes:

• no hook runs fm;
• /usr/bin/fm is called by its full path, on macOS only.

Audio transcripts (media=audio, section 4) were dropped as outside TokenEconomizer's scope. We'll consider them as a separate plugin. The results below are unchanged.

macOS 27 ships Apple's Foundation Models CLI at /usr/bin/fm. It runs a ~3B model ("AFM 3 Core") on the Neural Engine and GPU, with a 4,096-token window, guided JSON-schema output, image input, and no login or key. That made me ask where an on-device model helps TokenEconomizer, and where it doesn't. This issue covers the approach, the rationale, and every experiment, including the negative ones. Two options held up: the judge and audio transcripts. #185 adds the judge, opt-in. Audio was dropped in review (section 4).

Where an on-device model could plug in

All three of TokenEconomizer's model calls go through sidecall.runner_for(host, config) → runner(prompt, model) -> (text, effective_tokens):

Side call	What it asks	Fit for a 3B model with a 4k window	Result
Stuck judge	a ≤1,500-token digest → {stuck, confidence, reason}	a classification task, which suits guided output	✅ usable with a schema and a calibrated floor (below)
Redirect writer	the digest → 2–4 sentences	short constrained text	⚠️ follows every hard rule, but generic
Fact proposer	8 session digests → facts with Python-regex detectors and citations	long context plus authoring code	❌ no usable facts (below)

On Claude Code each side call is a claude -p call that carries the CLI's own base prompt. Your docs measured about 14k effective tokens for a real judge call, and about 25k for two verdicts plus a redirect. I measured 6,981 effective tokens per judge call on Haiku 5.5.

1. The judge

Data. I used 502 SWE-agent trajectories from the public nebius/SWE-agent-trajectories, 94 of them resolved. SWE-agent commands map onto TE's tool names: open/goto/scroll → Read, edit → Edit, create → Write, search_*/find_file → Grep, the rest → Bash. I replayed each trajectory call by call through loopguard.TurnGuard with default thresholds. The envelope is p90 of resolved lengths × 1.25 = 40 calls, shared because there are too few resolved runs per repo. A recording judge that always spares collected every digest the guard would have sent, up to 3 per trajectory: 550 points. I judged a stratified sample of 302 with llm_stuck_judge itself:

• 52 points from resolved runs, where a redirect interrupts a success.
• 150 from runs that ran out of their context budget, the likeliest loops.
• 100 from other unresolved runs.

Haiku went through TE's own claude_call; fm went through the PR's fm_call.

Judge	threshold	resolved ↓	exhausted	other	AUC (exhausted vs resolved)	tokens/call	s/call
Haiku 5.5, current default	0.70	26/52 (50%)	115/150 (77%)	53/100 (53%)	0.684	6,981	3.9
Haiku 5.5	0.85	10/52 (19%)	63/150 (42%)	21/100 (21%)	0.684	6,981	3.9
fm, verdict-first schema (the PR)	0.70	37/52 (71%)	123/150 (82%)	76/100 (76%)	0.639	0	2.0
fm, verdict-first schema (the PR)	0.90	7/52 (13%)	56/150 (37%)	13/100 (13%)	0.639	0	2.0
fm, evidence-first schema	0.70	48/52 (92%)	143/150 (95%)	89/100 (89%)	0.638	0	4.5
fm, evidence-first schema	0.90	11/52 (21%)	71/150 (47%)	19/100 (19%)	0.638	0	4.5
fm, free text (no schema)	0.70	4/52 (8%)	13/150 (9%)	2/100 (2%)	0.383	0	1.9

What I take from this:

• A schema is required. In free text fm rarely says stuck, and it doesn't separate loops from successes: 9% of exhausted points against 8% of resolved ones, AUC 0.38. In 182 of its 283 "not stuck" verdicts the reason contains a loop phrase such as "repeated identical failures" (a keyword count; it also catches some hedged reasons). The schema helps but doesn't remove this: verdict-first still has 38 of 66, evidence-first 5 of 16, Haiku 7 of 104 (mostly hedged, such as "repeated the same failures, but then shifted to reading the codebase"). fm respond --schema constrains decoding, so the verdict always parses.
• A floor is required. fm's confidences are quantized (0.65/0.72/0.85/0.92). At the guard's 0.7 nearly every "stuck" counts, so the PR adds local_judge_confidence=0.9, applied only when fm answered.
• At 0.9, fm is close to Haiku's own trade-off curve but slightly below it (AUC 0.64 vs 0.68). It redirects fewer resolved runs than Haiku at 0.85 (13% vs 19%) and catches fewer exhausted runs (37% vs 42%). I chose the verdict-first schema because it is the more conservative of the two ("protect task success") and twice as fast. Evidence-first, which writes progress and repetition signs before the verdict, catches more loops (47%) and contradicts itself less, at 21% on resolved runs. It is a one-line swap if you prefer that point.
• Haiku at the default 0.7 redirected half of the resolved runs' decision points too. With the benchmark-default thresholds the judge does a lot of the work. That is about the guard's settings, not about fm.
• Cost: the 302 Haiku calls were 2.1M effective tokens. On fm they are 0, and the call returns in about 2 s.

Live, Claude Code 2.1.319 with the branch (--plugin-dir, TE_LOCAL_MODEL=fm). I ran 16 Haiku sessions in a toy repo: solvable tasks, plus tasks whose tests contradict each other. The judge ran on fm ("local": "fm", _tokens: 0 in its verdicts), and nothing was charged to the ledger. Haiku 5.5 does not loop on its own: it notices contradictory tests within 3–6 turns and stops, even when told to keep trying. A scripted 8-cycle loop drew one early verdict, a spare at 0.65 after about 6 calls. The backoff then moved the next check past the turn's end. Re-judged on the turn's final digest, fm says stuck at 0.92.

2. The redirect writer (local_roles=judge,nudge, opt-in)

On stuck digests it never told the agent to stop, never named line numbers, and never invented identifiers. But it mostly repeats the prompt's own examples ("re-read the failing test's expectation") and gets details wrong: it says "same file" for two files, and talks about a "failing test" for a make loop. It is probably no better than rules.nudge_text, so it stays opt-in and the default role list is judge.

3. The proposer: not a fit

I ran TE's Distiller over 15 iterative/dvc trajectories converted to Claude Code transcripts. The prompt was 8,856 characters (2,780 fm tokens), so it fits.

• Haiku: one call, 42.5k effective tokens, 32 s. It proposed two facts consistent with the sessions (the package layout, and "scripts run against the working tree").
• fm, free text: invalid JSON. Its regex escaping broke.
• fm with a schema: valid JSON and valid regexes, but invented facts ("dvc build", "dvc lint", "pip install dvc"), with detectors that would never match.

local_roles=proposer exists for completeness, but I would not recommend it.

4. Audio prompts: a clear win, but not in the PR (media=audio)

Claude Code's Read rejects an audio file as binary before PreToolUse runs, so a Read hook cannot help. I verified that: the hook never fired. The prompt hook can. The task, with a Haiku 5.5 agent, 5 runs per arm and hidden tests: "the PM left a voice memo notes/bug_memo.aiff; implement exactly what it asks" (round half up, reject negative or >1 percentages).

Arm	pass	cost	effective tokens	turns
no plugin	0/5	$0.0060	54,269	12.8
Read hook (never fires for audio)	0/5	$0.0051	45,983	11.6
standalone prompt hook	5/5	$0.0041	35,483	7.6
#185's first version, TE_MEDIA=audio	5/5	$0.0035 (−42%)	30,306 (−44%)	7.6

The transcript contains recognition errors ("Refunds Dog Pie" for refunds.py, "sand" for "cent"). The agent read through them every time. The transcriber is Apple's SpeechAnalyzer/SpeechTranscriber: 38× realtime on an M2 in my tests, on-device, with no entitlement needed. It is a 40-line Swift helper compiled once with swiftc.

Review outcome. It isn't in TokenEconomizer, for two reasons:

• The plugin reads finished sessions to change what later sessions cost; transcribing files at prompt time is outside that.
• It would be the project's first compiled helper, which CI can't build or test.

The review also found two bugs in it:

• With repomap=on, the repository map replaced the transcript on the first prompt.
• Up to 3 files × 15 s of transcription could overrun the prompt hook's 20 s limit, losing the loop guard's new turn with it.

Both would need fixing in a separate plugin.

5. Images: no win, so not in the PR

I tried a PreToolUse(Read) hook that hands the agent fm's text transcription of a screenshot (updatedInput → a markdown file; a second Read of the same path gets the pixels). fm transcribed the test screenshots almost exactly. But cost didn't move, and every run passed:

Task (5 runs per arm)	base cost / eff tokens	transcription cost / eff tokens
debug from a traceback screenshot	$0.0039 / 33,817	$0.0037 / 32,648
match a pricing-table screenshot	$0.0032 / 26,593	$0.0032 / 27,709
four QA screenshots	$0.0099 / 83,865	$0.0096 / 85,235

A screenshot is a small share of a session, and the transcription is comparable in size. The hook also adds 3–6 s per image. So no image mechanism.

Limits

• Only on macOS 27 with Apple Intelligence on, Apple silicon. On other systems local_model=fm does nothing, and the judge stays on the agent's CLI.
• fm's window is 4,096 tokens. The judge's and redirect writer's prompts fit with room to spare: the largest of the 550 replayed digests, with the judge's instructions, is 1,359 fm tokens.
• The judge numbers are an offline replay on another agent's trajectories, not a rerun of the paper's arms, which need the Linux/Docker harness. A rerun of spiral-only with local_model=fm against Haiku on a few repos would be the decisive test. I'm glad to help with it on a Mac host if the harness can reach one.

Evidence

The scripts and raw per-point results are in an evidence bundle:

• The replay through TurnGuard.
• The judge harness that calls llm_stuck_judge.
• All 302 × 4 verdicts (Haiku; fm verdict-first, evidence-first and free text).
• The media A/B runner with its hidden tests (audio and images).
• The proposer runs.
• The live session log.

I'll post a link to the bundle in a comment on this issue.