Streaming /v1/messages: message_start reports input_tokens 0 and no cache fields

Abierto
  • ¿Algo no está bien?
#718 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
62/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
api, backend

Línea de trabajo

Start with anthropic_event_stream in python/freetoken/server/anthropic_api.py (around L475-482, where message_start is sent with zeros) and the GenDone handling near L582-588. Then read _generate_events_impl in python/freetoken/server/generation.py (around L702-704), where PromptAdmittedMsg counts are summed but never emitted. Run the CPU reproduction from the issue to see the current message_start output. Done when message_start carries the admitted input_tokens and cache_read_input_tokens for both cache_report settings, matching the final message_delta.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happened

A streamed /v1/messages response always opens with message_start.message.usage = {"input_tokens": 0, "output_tokens": 0} and no cache fields, even under --enable-cache-report. The real counts only arrive in the final message_delta. Anthropic's streaming format reports the prompt's input_tokens and cache fields in message_start, so a client that reads input usage there sees 0 prompt and 0 cache-read tokens on every streamed turn, while the non-streaming response reports the real values.

--enable-cache-report message_start usage (expected) message_start usage (got)
off input_tokens: 1200 input_tokens: 0
on input_tokens: 176, cache_read_input_tokens: 1024 input_tokens: 0 (no cache field)

Minimal reproduction (CPU only, PYTHONPATH=python):

import asyncio, json
from freetoken.server import anthropic_api as A
from freetoken.server.generation import ContentDelta, GenDone
async def ev():
    yield ContentDelta("hi")
    yield GenDone("stop", 1200, 5, cached_tokens=1024)
async def main():
    for cr in (False, True):
        async for f in A.anthropic_event_stream(ev(), "m", 1, cache_report=cr):
            d = json.loads(f.splitlines()[1][6:])
            if d["type"] == "message_start":
                print(f"cache_report={cr} message_start:", d["message"]["usage"])
            elif d["type"] == "message_delta":
                print(f"cache_report={cr} message_delta:", d["usage"])
asyncio.run(main())

cache_report=False message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=False message_delta: {'input_tokens': 1200, 'output_tokens': 5}
cache_report=True message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=True message_delta: {'input_tokens': 176, 'output_tokens': 5, 'cache_read_input_tokens': 1024}
Cause (current main, 9b68e007)
  • python/freetoken/server/anthropic_api.py:475-482: anthropic_event_stream sends message_start before it reads any event, with a hard-coded AnthropicUsage(input_tokens=0, output_tokens=0). The comment there says "input_tokens filled in by the terminal message_delta".
  • python/freetoken/server/anthropic_api.py:582-588: the real usage, built by _anthropic_usage (L417-428), is only sent on GenDone.
  • The counts are available long before GenDone. The scheduler sends PromptAdmittedMsg(prompt_tokens, cached_tokens) when it admits the prompt (python/freetoken/scheduler/scheduler.py:885). The tokenizer turns that into an empty ack (python/freetoken/tokenizer/server.py:43-51), and _generate_events_impl adds it up (python/freetoken/server/generation.py:702-704). But no GenEvent carries these counts out, so the Anthropic streamer cannot use them.
Proposed fix

Have _generate_events_impl emit a small event (for example PromptAdmitted(prompt_tokens, cached_tokens)) when the admission ack arrives. anthropic_event_stream would then send message_start on that event, filling usage through _anthropic_usage(prompt, 0, cached, cache_report), so its input_tokens/cache_read_input_tokens split matches the final message_delta. If a keepalive is due before admission, message_start would go out first with the current zeros, as it does today. The OpenAI and Responses streamers can ignore the new event.

Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 on an RTX 5080 16 GB; the reproduction above is CPU-only on main 9b68e007.

Lenguaje dominante
Python
Estrellas
14.2k
Forks
1.4k
Merge medio
18 h 46 min
PR fusionados (30 d)
98

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de FlashML-org/FreeToken

Todos los issues de FlashML-org/FreeToken

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.