Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug] Context overflow is checked only after tokenization in the worker: a streamed request gets 200 plus an in-stream error, and an oversized `max_tokens` is silently clamped

Aperta
  • Qualcosa non va?
#749 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
api, backend

Direzione di ricerca

Start with python/freetoken/server/generation.py and trace submit_generation, then inspect message/tokenizer.py, tokenizer/tokenize.py, and scheduler/scheduler.py. Reproduce the context boundary for a streamed request and for prompt + max_tokens exceeding the limit. Done means oversized requests receive HTTP 400 with context_length_exceeded before streaming, counted token IDs are reused by the worker, and the policy for omitted max_tokens is resolved.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

The context-length check runs only in the scheduler, after the worker has rendered and tokenized the prompt. Two
behaviours follow:

  1. A streamed request whose prompt does not fit gets HTTP 200, an assistant role chunk, and then an SSE
    {"error": ...} event, not a 400 context_length_exceeded. The prerender_error docstring describes the client
    side of this: some agents show only "empty response" for an in-stream error.
  2. A request whose prompt fits but whose prompt + max_tokens does not has its max_tokens lowered silently (a log
    line only). The client asked for N tokens, cannot know it got fewer, and receives finish_reason: "length" early.
    The scheduler's own error text already treats prompt plus generation as the limit ("... maximum (prompt +
    generation)").

We fixed both by tokenizing once in the frontend, before any response header, and passing those exact IDs to the
worker.

Other fixes of ours from the same list, checked against main 4d2e2e2: overlap-scheduler output budget (requests
stopped at 127 of 128 tokens) is fixed upstream by #546 (scheduler.py:353). Request-local seed is still
missing on main, but #693 already reports it (our notes go there as a comment). Our disconnect handling is posted as
comments on #644 and #567. Our in-engine abort changes apply only to our layer-major prefill and do not concern
upstream.

Setup

  • FreeToken 0.1.3 cac247a8 plus our fork; nvidia/Qwen3.6-35B-A3B-NVFP4; context 131072; OpenAI
    /v1/chat/completions
  • The behaviour is in the HTTP/scheduler layer; GPU and model do not matter.

Problem in upstream today (main 4d2e2e2)

  • python/freetoken/server/openai_api.py:194-199: for stream: true, prerender_error runs first, but
    server/generation.py:358-387 only renders the template (manager.render_prompt) and never counts tokens.
  • openai_api.py:201-210: submit_generation (generation.py:297-315) sends the raw messages to the worker and
    returns a uid. StreamingResponse is then returned, which commits 200.
  • openai_api.py:250-256: the stream yields the role chunk before any reply. If the worker then rejects the prompt,
    generation.py:700-701 raises GenerationError inside the stream, and openai_api.py:270-276 sends it as an SSE
    error event.
  • python/freetoken/scheduler/scheduler.py:514-539: the only length check (max_output_len <= 0), which produces the
    context_length_exceeded reply.
  • scheduler.py:540-544: if msg.sampling_params.max_tokens > max_output_len: it sets
    max_tokens = max_output_len and only logs a warning.
  • The prompt is rendered twice (frontend pre-check, then worker) and tokenized once, in the worker.

What we changed

  • freetoken/python/freetoken/server/generation.py (submit_generation): for text-only requests the frontend
    tokenizer (state.frontend_tokenizer) encodes the conversation once, in a worker thread. If count < 1 or
    count + max_tokens > max_seq_len, it raises GenerationError(..., code="context_length_exceeded"). Every adapter
    already turns that into a 400 before a StreamingResponse exists. Template errors become a 400 the same way.
  • message/tokenizer.py: TokenizeMsg.prepared_input_ids. tokenizer/tokenize.py uses those IDs as they are, so the
    worker does not render or tokenize again, and the IDs it serves are exactly the ones that were counted.
  • scheduler/scheduler.py: max_tokens > max_output_len now returns ErrorReplyMsg(code="context_length_exceeded")
    instead of clamping. The scheduler stays the authoritative check, for image requests and other paths.

Measured impact

Boundary gate on the 128K profile (results/context-gates/final-f1-c128-a1-evaluation.json):

  • prompt 130,396 tokens with max_tokens 676 (sum exactly 131,072) -> 200, finish_reason: "stop" after 28 tokens;
  • the same request plus one token -> 400 context_length_exceeded.

Not measured: the extra frontend tokenization time per request (the worker no longer tokenizes, so the total work is
roughly unchanged), and the 256K/64K profiles separately (their gates use the same check).

Correctness evidence

  • The worker serves exactly the IDs the frontend counted, so the admission count and usage.prompt_tokens cannot
    disagree.
  • Our release soak runs through this path (2-hour soak: 891 client requests, 0 errors; docs/RESULTS.md §3).

Limitations / not measured

  • Requests with images keep the old path (the frontend check is skipped and only the scheduler check applies).
  • When the client omits max_tokens, our check uses the default output budget (DEFAULT_MAX_OUTPUT_TOKENS, 32768,
    unless configured). A long prompt without max_tokens is then rejected where upstream would clamp. Upstream may
    prefer to clamp only an implicit default and reject only an explicit max_tokens.
  • Only /v1/chat/completions was exercised; /v1/completions, /v1/messages and /v1/responses share
    submit_generation, but we did not test them.

Proposal

A PR along the lines above: count tokens in the frontend before committing the stream, pass prepared_input_ids,
and reject (rather than clamp) an explicit max_tokens that does not fit. Open question: should an omitted
max_tokens keep today's clamp?

Our implementation, for reference (fork of 0.1.3, Apache-2.0; paths in the fork):

Lingua principale
Python
Stelle
14.2k
Fork
1.4k
Merge medio
21h 17m
PR unite (30g)
97

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FlashML-org/FreeToken

Tutte le issue di FlashML-org/FreeToken

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.