Context usage indicator counts base64 images as text and shows several times too many tokens

Aperta
  • Qualcosa non va?
#230 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Oscar-Williams ci sta già lavorando.

Dal 1/11/2026.

  • #232 di @Oscar-Williams — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
70/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
typescript
Ambito
ai, desktop

Direzione di ricerca

Start in src/utils/analyzeContext.ts: the estimate-only branch of countTokensWithFallback stringifies the whole conversation, and processAssistantMessage and processUserMessage stringify each block. Compare with roughTokenCountEstimationForMessages and roughTokenCountEstimationForBlock in src/services/tokenEstimation.ts, which already count images as a flat 2000 tokens. Done when a message with a large base64 image estimates to about 2000 tokens in a new src/utils/tests/analyzeContext.test.ts, and that test fails before the change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:desktop bug good first issue help wanted

What happens

The context usage indicator in the desktop app shows far more tokens than a session really uses when the conversation holds screenshots.

In one session (a model with a 1M-token window) the indicator showed 933,868 / 1,000,000, with 903,784 under "Messages". The provider's dashboard recorded about 140,000 input tokens for the latest request. Auto-compaction did not run. That is correct for the real usage, but it leaves the indicator looking wrong, and a nearly full bar tells the user to compact a session that is nowhere near its limit.

To reproduce

  1. Start a session in which the agent takes many browser screenshots, for example with the built-in browser tools (in the session above the agent had taken about 40).
  2. Open the context usage indicator below the input box.
  3. Compare "Messages" with the input tokens your provider reports for the latest request.

Cause

The desktop app reads the indicator through the sidecar, which asks the CLI for get_context_usage with estimateOnly: true (sessions.ts#L554, #L567).

In that mode countTokensWithFallback returns roughTokenCountEstimation(jsonStringify({ messages, tools })) (analyzeContext.ts#L85). It serializes the whole conversation, so the base64 data of every image is counted as text, at about one token per four characters. A screenshot of a few hundred KB of base64 becomes tens of thousands of tokens, where a provider charges about 1 to 2 thousand.

An image-aware estimator already exists. roughTokenCountEstimationForBlock counts an image or document as a flat 2000 tokens, and its comment explains that base64 must not go through JSON serialization (tokenEstimation.ts#L400). roughTokenCountEstimationForMessages uses it (tokenEstimation.ts#L327). The estimate-only path does not.

The per-category breakdown has the same problem: processAssistantMessage and processUserMessage stringify each block and estimate the string (analyzeContext.ts#L813, #L845), so "tool results" is inflated too.

A transcript from such a session holds 41 image blocks with about 5.9M base64 characters, which is about 1.48M tokens at one per four characters. That is enough to explain the inflated number. The CLI drops older tool results before estimating, so the indicator did not show the full 1.48M.

Why auto-compaction is not affected

shouldAutoCompact uses tokenCountWithEstimation (autoCompact.ts#L210, tokens.ts#L244). It starts from the real usage of the last API response and estimates only the messages added after it, so it saw about 140k and correctly did not compact.

Suggested fix

  • In the estimate-only branch, count the messages with roughTokenCountEstimationForMessages(messages) and the tools with a rough estimate of their JSON, instead of stringifying everything.
  • In processAssistantMessage and processUserMessage, estimate each block with the same image-aware per-block estimator (export it) instead of stringifying the block.
  • Add a test: a message holding a large base64 image estimates to about 2000 tokens, not hundreds of thousands. There is no test file for analyzeContext or tokenEstimation yet, so a new src/utils/__tests__/analyzeContext.test.ts is fine. Check that the test fails without the change.

The result stays an estimate and will not match the provider exactly. Making the indicator and auto-compaction agree by construction would mean anchoring the indicator on the last API usage as well. That is a bigger change and not needed to fix this.

Lingua principale
TypeScript
Stelle
490
Fork
47
Merge medio
8h 51m
PR unite (30g)
91

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di GoDiao/dreamcoder

Tutte le issue di GoDiao/dreamcoder

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.