DeepSeek V4.1 Flash (gb10): draft acceptance 24.5%/32.8% and draft depth locked to 5 — widths 2-5 reported by inspect but not servable

Aperta
  • Qualcosa non va?
#24 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

Start by reading the flare inspect and --graphs output alongside the serving flags and failure message in the issue; the payload does not name source files or tests. Determine whether reported draft widths are intended to be servable and whether a shallower verification prefix is supported. Done means documenting or fixing the mismatch and clarifying the supported controls for draft depth and sampling.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:scheduling bug model:deepseek-v4.1 performance priority:P2 triage:investigate

Environment

  • 4× DGX Spark (GB10 / SM121), TP4 over a RoCE ring (--roce-ring)
  • flare-inference 0.1.0 (PPA build, git 20261022044608.27aba893)
  • Package: local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revision gb10, file deepseek-v41-flash-1017-tp4-c16-dspark5.safetensors
  • Serving flags: --max-concurrency 16 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB

What I measured

Single stream, 200 generated tokens, computed from /metrics deltas:

workload steps speculative steps proposes/round accepted acceptance tok/step ms/step tok/s
prose 150 81 (54%) 5 118 24.5% 1.33 27.1 49.1
code 137 67 (49%) 5 132 32.8% 1.46 24.6 59.3

flare_speculative_rounds_total{width=...} is essentially only width="6" (575 rounds at batch=1, 7 rounds at width=1). Widths 2-5 are always zero.

Two more data points:

  • Disabling speculation entirely (--mtp-draft-tokens 0) gives prose 38.8 tok/s and code 39.2 tok/s, so speculation is a +70% / +178% win. The feature works; acceptance is just low.
  • Per-step cost with and without speculation is nearly identical (27.1 vs 26.9 ms/step on prose), so the 6 verification rows are essentially free at batch=1.

The issue

flare inspect reports:

speculation   mtp · widths 1, 2, 3, 4, 5, 6

but --graphs lists packed graphs only at depth 6:

verification16x6   verification  16 6 96
verification8x6    verification   8 6 48
verification4x6    verification   4 6 24
verification2x6    verification   2 6 12
speculative16x1    speculative   16 1 16

So widths 2-5 are reported but do not appear to be servable. Trying to use them fails:

Error: Execution(Manifest(Failure { message: "packed verification requires adaptive execution at the captured draft depth" }))

--no-mtp-adaptive-verification fails the same way. The only working settings are depth 5 with adaptive verification, or depth 0.

Questions

  1. Does the widths 1, 2, 3, 4, 5, 6 line in flare inspect describe usable serving widths, or only the draft depths the proposal head can produce? If the former, this package looks like it is missing the packed graphs for depths 2-5.

  2. Is there a supported way to serve at a shallower effective depth without re-exporting? --mtp-verification-token-budget documents "adaptive prefix selection at the captured draft depth", which reads like it could verify a shorter prefix inside the x6 graph. Is that its intended use, and does it change how many drafts are proposed, or only how many are verified?

  3. Is draft-head sampling tunable at all? For the same prompt, temperature: 0 and temperature: 1 produce byte-identical output (each run twice), and flare inspect reports temperature 1 fixed, top_p 1 fixed. That suggests the sampler is fixed at serve time. If acceptance is sensitive to how sharp the draft distribution is, is there an export-side flag for it?

  4. Is 24.5% (prose) / 32.8% (code) acceptance in the expected range for the gb10 DeepSeek V4.1 Flash package at depth 5, or does it suggest a preparation issue? For reference, the DSpark implementation of the same model in our SGLang setup exposes a draft temperature, block verification and a verification confidence cap. I have not found equivalents on the Flare side, so I cannot tell whether the gap is configuration or inherent to this package.

Happy to run any specific measurement or diagnostic you would like.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.