Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

seed and chat logprobs are silently ignored (200 with no effect); proposal for per-request seeded sampling

Aperta
  • Qualcosa non va?
#693 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
api, machine-learning

Direzione di ricerca

Start in server/api_models.py, where ChatCompletionRequest accepts unknown fields via extra="allow", then openai_api.py around line 658 where CompletionRequest rejects logprobs. Next, engine/sample.py sample_impl() and kernel/triton/sampling.py _gen_u show where seed is dropped. A small first step is returning 400 for unsupported seed or logprobs on the chat route. Full seeded sampling is a design task to agree on with maintainers first, and it depends on #224 for logprobs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happened

Two OpenAI request fields are ignored without an error, so the client gets a 200 response that does not do what it asked for:

Request Expected Got
/v1/chat/completions, temperature: 1.0, seed: 1234, the same request twice the same output different output
/v1/completions, temperature: 1.0, seed: 7, the same request twice the same output different output
/v1/chat/completions, logprobs: true, top_logprobs: 3 choices[].logprobs.content[] "logprobs": null, no error
/v1/completions, logprobs: 2 logprobs, or a 400 400 logprobs is not supported (correct, but it shows the gap)

temperature: 0 gives the same output each time in my tests (single short requests, thinking off). Other sampling fields work: top_p, top_k, min_p, the penalties, stop and max_tokens.

Minimal reproduction:

for i in 1 2; do curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen3.8-Flash-Next", "max_tokens": 20, "temperature": 1.0, "seed": 1234,
  "chat_template_kwargs": {"enable_thinking": false},
  "messages": [{"role": "user", "content": "Invent a name for a cat."}]}' | jq -r '.choices[0].message.content'; done
Cause (current main, 0bca88f7)
  • server/api_models.py: ChatCompletionRequest has model_config = ConfigDict(extra="allow") and no seed or logprobs field. Both are accepted and then dropped. CompletionRequest also has no seed, but it rejects logprobs (openai_api.py around line 658).
  • SamplingParams has no seed, and engine/sample.py sample_impl() calls the sampling functions without seed / offset. The Triton sampler (kernel/triton/sampling.py, _gen_u) already accepts seed and offset, but only one value for the whole batch.
Proposed fix

seed:

  1. Add seed: int | None to ChatCompletionRequest, CompletionRequest and SamplingParams, and carry it to the scheduler like the other sampling fields.
  2. In Sampler.prepare(), build a per-row (seed, offset) pair for the rows that have a seed. Use the number of generated tokens as the offset, so each step draws a new number but a replay draws the same one.
  3. Make the uniform draw per row: in the Triton path, tl.rand(seed_row, offset_row) (Philox) inside the kernel, or one generator per seeded row in _gen_u. Rows without a seed keep the current unseeded draw. A batch with no seeded rows keeps exactly the current path. If the FlashInfer sampling path cannot take per-row seeds, send batches that contain a seeded row to the Triton path.
  4. Document the limit: a seed makes the draw reproducible, but the logits can still differ between batch compositions (see #307 for the near-tie case at temperature 0). So the guarantee is "best effort", like OpenAI's own seed.

logprobs:

#224 implements logprobs / top_logprobs for both routes, with raw pre-temperature values and no cost when not requested. It has been rebased and tested by several people. Merging it closes this part. Until then, /v1/chat/completions should return 400 for logprobs: true, like /v1/completions already does, and not null.

In general: extra="allow" on the request models turns every unsupported OpenAI field into a silent no-op. A short deny-list of fields that change the result (seed, logprobs, top_logprobs, logit_bias, n (see #682), response_format) that returns 400 until each one is implemented would make these gaps visible to clients.

Environment
  • Built from source: upstream main 2d8b9cc1 plus a local deploy stack. I checked each code path above on current main 0bca88f7, and none of it has changed since.
  • Ubuntu 24.04, Python 3.12, 2x RTX 6000 Ada 48 GB (driver 595.91), TP=2, 503 GB RAM.
  • Checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4.
  • All results above are reproducible directly against ft serve, with no proxy.
Lingua principale
Python
Stelle
14.2k
Fork
1.4k
Merge medio
21h 17m
PR unite (30g)
97

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FlashML-org/FreeToken

Tutte le issue di FlashML-org/FreeToken

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.