Context usage indicator counts base64 images as text and shows several times too many tokens

Abierto
  • ¿Algo no está bien?
#230 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Oscar-Williams ya está trabajando en esto.

Desde el 1/11/2026.

  • #232 de @Oscar-Williams — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
70/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
typescript
Área
ai, desktop

Línea de trabajo

Start in src/utils/analyzeContext.ts: the estimate-only branch of countTokensWithFallback stringifies the whole conversation, and processAssistantMessage and processUserMessage stringify each block. Compare with roughTokenCountEstimationForMessages and roughTokenCountEstimationForBlock in src/services/tokenEstimation.ts, which already count images as a flat 2000 tokens. Done when a message with a large base64 image estimates to about 2000 tokens in a new src/utils/tests/analyzeContext.test.ts, and that test fails before the change.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:desktop bug good first issue help wanted

What happens

The context usage indicator in the desktop app shows far more tokens than a session really uses when the conversation holds screenshots.

In one session (a model with a 1M-token window) the indicator showed 933,868 / 1,000,000, with 903,784 under "Messages". The provider's dashboard recorded about 140,000 input tokens for the latest request. Auto-compaction did not run. That is correct for the real usage, but it leaves the indicator looking wrong, and a nearly full bar tells the user to compact a session that is nowhere near its limit.

To reproduce

  1. Start a session in which the agent takes many browser screenshots, for example with the built-in browser tools (in the session above the agent had taken about 40).
  2. Open the context usage indicator below the input box.
  3. Compare "Messages" with the input tokens your provider reports for the latest request.

Cause

The desktop app reads the indicator through the sidecar, which asks the CLI for get_context_usage with estimateOnly: true (sessions.ts#L554, #L567).

In that mode countTokensWithFallback returns roughTokenCountEstimation(jsonStringify({ messages, tools })) (analyzeContext.ts#L85). It serializes the whole conversation, so the base64 data of every image is counted as text, at about one token per four characters. A screenshot of a few hundred KB of base64 becomes tens of thousands of tokens, where a provider charges about 1 to 2 thousand.

An image-aware estimator already exists. roughTokenCountEstimationForBlock counts an image or document as a flat 2000 tokens, and its comment explains that base64 must not go through JSON serialization (tokenEstimation.ts#L400). roughTokenCountEstimationForMessages uses it (tokenEstimation.ts#L327). The estimate-only path does not.

The per-category breakdown has the same problem: processAssistantMessage and processUserMessage stringify each block and estimate the string (analyzeContext.ts#L813, #L845), so "tool results" is inflated too.

A transcript from such a session holds 41 image blocks with about 5.9M base64 characters, which is about 1.48M tokens at one per four characters. That is enough to explain the inflated number. The CLI drops older tool results before estimating, so the indicator did not show the full 1.48M.

Why auto-compaction is not affected

shouldAutoCompact uses tokenCountWithEstimation (autoCompact.ts#L210, tokens.ts#L244). It starts from the real usage of the last API response and estimates only the messages added after it, so it saw about 140k and correctly did not compact.

Suggested fix

  • In the estimate-only branch, count the messages with roughTokenCountEstimationForMessages(messages) and the tools with a rough estimate of their JSON, instead of stringifying everything.
  • In processAssistantMessage and processUserMessage, estimate each block with the same image-aware per-block estimator (export it) instead of stringifying the block.
  • Add a test: a message holding a large base64 image estimates to about 2000 tokens, not hundreds of thousands. There is no test file for analyzeContext or tokenEstimation yet, so a new src/utils/__tests__/analyzeContext.test.ts is fine. Check that the test fails without the change.

The result stays an estimate and will not match the provider exactly. Making the indicator and auto-compaction agree by construction would mean anchoring the indicator on the last API usage as well. That is a bigger change and not needed to fix this.

Lenguaje dominante
TypeScript
Estrellas
490
Forks
47
Merge medio
8 h 51 min
PR fusionados (30 d)
91

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de GoDiao/dreamcoder

Todos los issues de GoDiao/dreamcoder

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.