[Engine] serve: an asked include_usage never gets the usage chunk of its own; spec-following clients count 0 tokens

Open Beginner friendly
  • Is something wrong?
#5,448 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
api, backend

Research direction

Start the mock server with python serve/server.py --engine mock --port 8096 and reproduce with the curl request in the issue. Trace the streaming chat completion response where stream_options.include_usage is handled; the issue does not name a source file or test. Done when the final usage event has empty choices and carries the usage block, as in the expected response.

Written by the indexing model from the issue text.

Description

Problem: a streamed completion asked for stream_options.include_usage never gets
OpenAI's final usage chunk of its own. The usage rides on the finish chunk, whose
choices stay filled (delta {}, finish_reason stop). The contract's last chunk carries
empty choices with the usage block, and llama.cpp's server answers that way. A client
that reads usage from that chunk - the shape the OpenAI docs and llama.cpp clients
expect - sees no usage at all and falls back to estimating.

Environment: Strata 0.2.10 (commit ffa80d73, engine 0.1.56) and current main
(9f7d977d), Linux x86_64. The serving engines run strata with
Qwen3.8-Flash-Next IQ3_S, 262k context. The repro needs no GPU and no model - it
uses the repository's own mock engine.

Reproduce (no GPU, about ten seconds):

  cd Strata        # a plain, unmodified checkout
  python serve/server.py --engine mock --port 8096 &
  curl -sN localhost:8096/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model":"m","messages":[{"role":"user","content":"hi"}],"stream":true,"stream_options":{"include_usage":true}}' | grep '^data:' | tail -2

Expected: the second-to-last event is a chunk with "choices": [] carrying the usage
block - OpenAI's streaming contract, as llama.cpp serves it.

Actual: no choice-less chunk is ever sent. Observed tail (ffa80d73, mock engine,
shortened):

  data: {... "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}], "usage": {"prompt_tokens": ...
  data: [DONE]

Effect on us, measured on two serving hosts: a gateway that counts completion tokens
only from the spec chunk (ours injects include_usage towards the host
and drops the chunk again for clients that did not ask) accounts every stream from
its character estimate instead. Tool-call-only turns - most of an agent's turns -
carry no content characters, and the estimate gives 0. In a 24 h window, 204 of 310
completion rows for this model logged completion_tokens 0 while the engine's own
DONE lines reported roughly 100 to 1300 generated tokens each; the per-model speed
column fell to about 2 tok/s against a real 40-45 tok/s decode. With a local eight-
line patch that adds the spec chunk, the same gateway logs exact prompt and completion
counts on every row and its usage-reported flag flips.

Reproduction status: reproduced locally - mock engine at two revisions, live serve
on two hosts.

Dominant language
C++
Stars
11.6k
Forks
1k
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Niko1221/Strata

All issues in Niko1221/Strata

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.