DeepSeek V4.1 Flash TP3 on GB10: a speculative round costs 1.4-2x a width-one round, a fixed draft cost plus about 4 ms per verified row

Aperta
  • Qualcosa non va?
#153 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
machine-learning, performance

Direzione di ricerca

Start by tracing the routed MoE kernel selection and benchmark path for the reported 4-6 row verification shapes; the issue names MoETinyDecodeKernelBackend and SiluMoEDynamicKernelSilu but gives no source files or tests. Compare kernel timings and selection under the stated setup, then determine whether a faster small-M path fits b12x. Done means a measured improvement to speculative round cost on this setup.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:kernels area:scheduling model:deepseek-v4.1 performance triage:design-review

Setup

  • Three DGX Spark GB10 hosts, DeepSeek V4.1 Flash TP3, current main (921afd5c). Packages exported with --max-concurrency 4 and --mtp-draft-tokens 5, and a second one with --mtp-draft-tokens 3; served with --scheduler-policy throughput.
  • One stream, streamed chat completions, temperature 0, thinking off, 512 output tokens. Per round, flare_speculative_gpu_nanoseconds_total matches the wall time, and readiness waits are 0.4-2.2 ms, so the round cost is GPU replay.
  • Kernel times come from nsys launch --cuda-graph-trace=node on rank 0 during one 200-token prose request per package (and one with --mtp-draft-tokens 0 served on all ranks).

Round cost

ms per round tokens per round
width 1 (--mtp-draft-tokens 0) 27.5 1
width 4 (3 drafts) prose 42-46, code 46 prose 1.8-2.4, code 3.1-3.7
width 6 (5 drafts) prose 46-50, code 53-55 prose 2.0-2.5, code 3.9-5.0

Where the extra time goes

Routed MoE per layer on rank 0:

1 row 4 rows 6 rows
target layers (40, top-6) 164 µs, MoETinyDecodeKernelBackend (two phases) 419 µs, SiluMoEDynamicKernelSilu 433 µs, SiluMoEDynamicKernelSilu
draft layers (3, top-3) not run 188 µs, MoETinyDecodeKernelBackend 308 µs, SiluMoEDynamicKernelSilu
  • Over the 40 target layers, the routed MoE goes from about 6.6 ms per round at width one to about 17 ms at widths 4 and 6.
  • The dynamic kernel takes almost the same time for 4 and 6 rows, although 6 rows can touch up to 36 experts and 4 rows up to 24.
  • For the 4-row top-6 shape, the default-effort race measured the micro candidate at 707 µs against 582 µs for the winning dynamic configuration (cold-cache protocol), so the selection follows the measurement. The draft's 4-row top-3 layers do run the tiny-decode kernel.
  • One dense GEMM (grid (1,1,80)) takes 114 µs per layer in the width-6 round against 21 µs at width one, but 99 % of it overlaps the routed MoE on the other stream, so it does not lengthen the round.
  • _markov_step adds about 1.7 ms per round, and a BF16 GEMV that takes 6 µs at width one takes 188 µs at width six (about 1.8 ms per round).

Question

On this setup one-stream decode is about 45 tok/s on prose and 80 on code, and most of the gap between a width-one and a speculative round is the routed MoE at 4-6 rows. Is a faster small-M routed-MoE path for these verification shapes in scope for b12x? I can collect more data if useful: other batch sizes, --autotune-effort audit, or ncu on the dynamic kernel.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.