Proposal: native BF16 MTP speculative decoding for dense Qwen3.8-27B (qwen38_dense), with results on SM120

Abierto
  • ¿Algo no está bien?
#20 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
15/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
infrastructure

Línea de trabajo

Read the offered branch, starting with commit febc02f and the existing speculative sampling, verification, commit, and GDN recovery paths named in the proposal. Reproduce the SM120 checkpoint and restore checks and the paired decode measurements on the pinned checkpoint. Done means maintainers decide whether to add dense Qwen3.8-27B MTP, which default budget to use, and whether the all-NVFP4 checkpoint belongs in scope.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

model:qwen3.8-27b triage:design-review
What

Both pinned Qwen3.8-27B NVFP4 checkpoints ship a native MTP layer: 15 BF16 mtp.* tensors, 849,398,784 bytes,
mtp_num_hidden_layers = 1, with embedding and head shared with the target. We composed it into qwen38_dense
using existing mechanisms only:

  • existing speculative sampling and commit (PreparedSpeculativeDistributions / PreparedStochasticVerification /
    PreparedSpeculativeCommit);
  • existing compact GDN recovery records and commit, plus _commit_convolution;
  • the existing draft page-table view (page_offset = 1) and lag-1 draft cursor;
  • width-1 ordinary decode.

It adds no new graph-operation family, Protobuf field or native policy.

Conventions (verified against vLLM qwen3_5_mtp.py / llm_base_proposer.py and SGLang qwen3_5_mtp.py)
  • fc(cat(pre_fc_norm_embedding(embed(x_{p+1})), pre_fc_norm_hidden(h_p))), with all Qwen norms zero-centered (1+w).
  • The target feedback h_p is post-final-norm, and recursive feedback is the post-mtp.norm output.
  • RoPE position p for the pair (x_{p+1}, h_p). The draft cache covers consumed-1 pairs.
  • Prefill and round absorption write target-derived pairs, and drafting starts at consumed-1.
Results (RTX PRO 4500 Blackwell SM120, 32 GB, stock 200 W)
  • Teacher-forced vs an independent PyTorch MTP, official checkpoint, 383 rows crossing pages: KL mean 0.00057,
    top-1 agreement 99.2%, top-10 overlap 99.0%, repeat runs bit-exact.
  • Native exported package, 4 concurrent requests: 216/216 (plain, fixed-3 and adaptive-3) on the official
    checkpoint. On the third-party all-NVFP4 checkpoint, 216/216 with prefix-cache hits (plain, fixed-7, adaptive-7).
    Both runs include mixed budgets, padded lanes, grammar and logprobs. Both require the attention-tail fix
    (#14, local-inference-lab/flashinfer#43).
  • Checkpoint/restore: exact draft K/V and target state across 13 page-boundary prefixes, with width transitions
    W8→W1→W3→W2→W8.
  • Speed, one request at a time (median native decode tokens/s, paired against plain in the same package):
Checkpoint Preset Plain Fixed-3 Adaptive Speedup
Official, 512 tokens Thinking ~37.5 78.9-81.2 79-82 (adaptive-3) ~2.0-2.2×
All-NVFP4, 1024 tokens Thinking ~45 87.5-88.4 84.8-86.3 (adaptive-7) ~1.9-2.0×
  • At 65K context, decode is about 2× faster but end-to-end only about 1.15×, because prefill bounds it.
  • At 4 concurrent requests we don't quote a speedup factor. The plain arm shows an unexplained "3+1" admission
    wave (one lane starts about 6 s late) that inflates the ratios. We'll file it separately once it's reproduced on
    an unmodified upstream package.
Limitations we're stating up front
  • The throughput runs use fixed-budget outputs that are truncated in the thinking phase. There's no
    completed-answer quality or statistical-equivalence claim.
  • A small authored-case suite shows the same automatic-check failures per case in both arms, and one manually
    observed MTP answer error. That's neither evidence of regression nor of equivalence.
  • Cross-width target scores differ slightly (KL ≈ 0.01) and are bit-exact within each width, so output isn't
    identical to plain decoding.
  • Only SM120 was tested: no SM121, no TP.
  • qualify/ speculative tools are Flash-Next-specific, so measurements used an ad-hoc paired native client. There is
    no replay-floor or Nsight operator attribution.
Commits offered (branch Maxrio9:qwen38-dense-native-mtp, all Codex-authored with design-review acks)
  • febc02f dense native MTP;
  • 472b862 per-request prepared-scratch cost field;
  • b2f7e51 serialized tensor-FP8 input reuse (memory: 4.31 GiB → 88 MiB quantized-input storage per 4096-token
    forward, needed for the official MTP catalog to fit 32 GB);
  • f3713e0 strided NVFP4 GDN staging, needed only for the third-party all-NVFP4 checkpoint. We can drop or split it
    if only the pinned checkpoint is wanted.

The independent fixes are separate PRs: #16 (snapshot pages), #18 (pinned export), #15 (docs) and #19 (qualify schema). The branch also contains them, so it is self-contained.

Question for maintainers

Is dense Qwen3.8-27B MTP wanted in qwen38_dense? If so, should the default be adaptive-3 (current) or fixed-3,
and should the all-NVFP4 checkpoint be in scope?

Disclosure

This work was written by AI coding agents: Codex (GPT-6) wrote the code, and Claude Code reviewed the MTP math, state contracts and test harnesses and filed this issue. Claude Opus 5.5 didn't author or review any kernel, fusion, packing, overlap or scratch changes. Everything ran on my hardware: RTX PRO 4500 Blackwell, SM120, driver 615.78.08 open module, CUDA 13.4.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.