Counting savings from request-rewriting plugins (context-guru, tool-prune): open questions

This is an open design discussion with no code change requested yet. Read core/cost/settle/avoided.go to see how tool-prune savings are priced today, then dash/event.go in the context-guru proxy for its savings model. Done means the open questions have a written, agreed direction with the context-guru team, not a patch.

Cortex records what every LLM request costs, and a growing set of plugins, context-guru and tool-prune among them, rewrite requests so they cost less. We would like Cortex to report the savings those plugins deliver in a way operators can rely on. Getting that right touches both projects, so we would like to work it out together with the context-guru team.

This issue collects the open questions as we currently understand them. It deliberately stops short of proposing a design, so we can start from a shared picture of the problem. Corrections and additions are very welcome, especially where we have misread how context-guru works.

Terms

• Spend: what a request actually cost. Cortex prices it from the token counts in the provider's response.
• Saving: what a request would have cost without a plugin, minus what it did cost. The request that was never sent cannot be measured, so every saving rests on a model. We call that model the baseline.
• Cache tiers: providers bill prompt tokens differently depending on the prompt cache. A cache read costs about 0.1x the normal input rate and a cache write about 1.25x, so where a token lands can change its price by 12.5x.
• Overhead: what a plugin spends itself to produce a saving, for example its own LLM calls.

Where things stand

1. Spend is priced in one place, core/cost/settle, from Cortex's rate tables, and written to the cost ledger.
2. Savings are priced in the same place, today for tool-prune only. tool-prune publishes plain facts: bytes removed, body size, and whether the change was applied. settle.Avoided (core/cost/settle/avoided.go) converts bytes to tokens using the request's own prompt-tokens-per-byte ratio, prices them at the cache tier the request landed in, and marks the figure as an estimate. Plugins join this record through an explicit list (savingComponents).
3. The context-guru plugin in Cortex does not report savings yet. It rewrites the body and logs tokens before and after, but Cortex has nothing to price, so its contribution currently shows as $0.
4. context-guru's standalone proxy already has a thoughtful savings model (dash/event.go, Event.Price). It separates content removed for the first time from content that is only re-sent on later turns, adjusts tokenizer counts to what providers bill, and nets out its own LLM calls and keep-alive pings. It has worked through several of the questions below in practice.

Open questions

1. Keeping one set of numbers. Spend is priced from Cortex's rate tables, and context-guru keeps its own. If both price parts of the same request, one request can end up with figures based on different rates. Rate tables drift easily (Cortex has had its own share of missing or stale model entries), so it would help to agree on a single source.

2. A figure operators can trust. Because a saving is a counterfactual, its value depends on the model behind it. We would like an operator reading "saved $X" to know how it was computed and who stands behind it. That holds for any plugin, Cortex's own included.

3. Effects beyond bytes removed. Cortex's current model knows only "N bytes removed, priced at this request's tier". Several things context-guru does fall outside that:

   • Cache effects. Rewriting earlier history can change whether a request reads from the prompt cache or writes to it. Spend is still recorded correctly. The saving, though, is priced at the tier the rewritten request landed in. When a rewrite causes a cache write, the saving is therefore priced at the write rate, and it can look largest on exactly the requests where the rewrite cost the most.
   • The plugin's own LLM calls. The summarizer, and the reuse of the agent's own model (model.source: incoming), go out through core/llmclient. By design that client does not pass through the Cortex listener, so these calls are not in the ledger. The ibac judge uses the same client, so this is not specific to context-guru.
   • Keep-alive (context-guru#381). A ping costs a little now to avoid a cache rebuild later. The cost and the benefit land on different requests, and the benefit only materializes if a real request arrives after the cache would otherwise have expired.

4. Content re-sent every turn. Agents re-send the whole conversation each turn, so content removed once appears removed on every later turn. context-guru's dashboard notes that pricing each turn naively overcounts by 13.1x on one measured window. For tool-prune, pricing every turn is right, because the tools really would have been re-sent each time. For history compaction it is not. Telling the two apart requires knowing which removed content is new this turn, and today only the context-guru engine has that information.

5. Two ways of counting tokens. Cortex estimates tokens from a byte ratio calibrated on each response. context-guru counts with a tokenizer (o200k_base) and applies a correction factor per model family. The same removal can come out as different token counts.

6. Credit across several plugins. Request rewriters run in a chain: tool-prune and context-guru can both shrink one request, and each sees what the previous one left. The framework records the body the client sent, the body that went upstream, and which plugins wrote it. It does not record how much each plugin removed, so splitting credit between them is still open.

7. Observe mode. Under on_error: observe a plugin's change is computed but not applied. Any saving from it is a "would have saved" figure that should stay separate from money actually not spent. tool-prune marks these as Projected, and the same distinction will matter for any new source of savings.

Questions we would like to explore together

• What should an operator be able to rely on when they see a savings figure?
• Which facts only the context-guru engine can know, and which can Cortex observe directly?
• How should overhead (a plugin's own calls, keep-alive pings) appear next to the savings it pays for?
• Where should the baseline model live, and how should changes to it be reviewed?

Looking forward to the discussion.