DeepSeek V4.1 Flash (gb10): draft acceptance 24.5%/32.8% and draft depth locked to 5 — widths 2-5 reported by inspect but not servable

Abierto
  • ¿Algo no está bien?
#24 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
machine-learning

Línea de trabajo

Start by reading the flare inspect and --graphs output alongside the serving flags and failure message in the issue; the payload does not name source files or tests. Determine whether reported draft widths are intended to be servable and whether a shallower verification prefix is supported. Done means documenting or fixing the mismatch and clarifying the supported controls for draft depth and sampling.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:scheduling bug model:deepseek-v4.1 performance priority:P2 triage:investigate

Environment

  • 4× DGX Spark (GB10 / SM121), TP4 over a RoCE ring (--roce-ring)
  • flare-inference 0.1.0 (PPA build, git 20261022044608.27aba893)
  • Package: local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revision gb10, file deepseek-v41-flash-1017-tp4-c16-dspark5.safetensors
  • Serving flags: --max-concurrency 16 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB

What I measured

Single stream, 200 generated tokens, computed from /metrics deltas:

workload steps speculative steps proposes/round accepted acceptance tok/step ms/step tok/s
prose 150 81 (54%) 5 118 24.5% 1.33 27.1 49.1
code 137 67 (49%) 5 132 32.8% 1.46 24.6 59.3

flare_speculative_rounds_total{width=...} is essentially only width="6" (575 rounds at batch=1, 7 rounds at width=1). Widths 2-5 are always zero.

Two more data points:

  • Disabling speculation entirely (--mtp-draft-tokens 0) gives prose 38.8 tok/s and code 39.2 tok/s, so speculation is a +70% / +178% win. The feature works; acceptance is just low.
  • Per-step cost with and without speculation is nearly identical (27.1 vs 26.9 ms/step on prose), so the 6 verification rows are essentially free at batch=1.

The issue

flare inspect reports:

speculation   mtp · widths 1, 2, 3, 4, 5, 6

but --graphs lists packed graphs only at depth 6:

verification16x6   verification  16 6 96
verification8x6    verification   8 6 48
verification4x6    verification   4 6 24
verification2x6    verification   2 6 12
speculative16x1    speculative   16 1 16

So widths 2-5 are reported but do not appear to be servable. Trying to use them fails:

Error: Execution(Manifest(Failure { message: "packed verification requires adaptive execution at the captured draft depth" }))

--no-mtp-adaptive-verification fails the same way. The only working settings are depth 5 with adaptive verification, or depth 0.

Questions

  1. Does the widths 1, 2, 3, 4, 5, 6 line in flare inspect describe usable serving widths, or only the draft depths the proposal head can produce? If the former, this package looks like it is missing the packed graphs for depths 2-5.

  2. Is there a supported way to serve at a shallower effective depth without re-exporting? --mtp-verification-token-budget documents "adaptive prefix selection at the captured draft depth", which reads like it could verify a shorter prefix inside the x6 graph. Is that its intended use, and does it change how many drafts are proposed, or only how many are verified?

  3. Is draft-head sampling tunable at all? For the same prompt, temperature: 0 and temperature: 1 produce byte-identical output (each run twice), and flare inspect reports temperature 1 fixed, top_p 1 fixed. That suggests the sampler is fixed at serve time. If acceptance is sensitive to how sharp the draft distribution is, is there an export-side flag for it?

  4. Is 24.5% (prose) / 32.8% (code) acceptance in the expected range for the gb10 DeepSeek V4.1 Flash package at depth 5, or does it suggest a preparation issue? For reference, the DSpark implementation of the same model in our SGLang setup exposes a draft temperature, block verification and a verification confidence cap. I have not found equivalents on the Flare side, so I cannot tell whether the gap is configuration or inherent to this package.

Happy to run any specific measurement or diagnostic you would like.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.