DeepSeek V4.1 Flash TP3 on GB10: a speculative round costs 1.4-2x a width-one round, a fixed draft cost plus about 4 ms per verified row

Abierto
  • ¿Algo no está bien?
#153 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
machine-learning, performance

Línea de trabajo

Start by tracing the routed MoE kernel selection and benchmark path for the reported 4-6 row verification shapes; the issue names MoETinyDecodeKernelBackend and SiluMoEDynamicKernelSilu but gives no source files or tests. Compare kernel timings and selection under the stated setup, then determine whether a faster small-M path fits b12x. Done means a measured improvement to speculative round cost on this setup.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:kernels area:scheduling model:deepseek-v4.1 performance triage:design-review

Setup

  • Three DGX Spark GB10 hosts, DeepSeek V4.1 Flash TP3, current main (921afd5c). Packages exported with --max-concurrency 4 and --mtp-draft-tokens 5, and a second one with --mtp-draft-tokens 3; served with --scheduler-policy throughput.
  • One stream, streamed chat completions, temperature 0, thinking off, 512 output tokens. Per round, flare_speculative_gpu_nanoseconds_total matches the wall time, and readiness waits are 0.4-2.2 ms, so the round cost is GPU replay.
  • Kernel times come from nsys launch --cuda-graph-trace=node on rank 0 during one 200-token prose request per package (and one with --mtp-draft-tokens 0 served on all ranks).

Round cost

ms per round tokens per round
width 1 (--mtp-draft-tokens 0) 27.5 1
width 4 (3 drafts) prose 42-46, code 46 prose 1.8-2.4, code 3.1-3.7
width 6 (5 drafts) prose 46-50, code 53-55 prose 2.0-2.5, code 3.9-5.0

Where the extra time goes

Routed MoE per layer on rank 0:

1 row 4 rows 6 rows
target layers (40, top-6) 164 µs, MoETinyDecodeKernelBackend (two phases) 419 µs, SiluMoEDynamicKernelSilu 433 µs, SiluMoEDynamicKernelSilu
draft layers (3, top-3) not run 188 µs, MoETinyDecodeKernelBackend 308 µs, SiluMoEDynamicKernelSilu
  • Over the 40 target layers, the routed MoE goes from about 6.6 ms per round at width one to about 17 ms at widths 4 and 6.
  • The dynamic kernel takes almost the same time for 4 and 6 rows, although 6 rows can touch up to 36 experts and 4 rows up to 24.
  • For the 4-row top-6 shape, the default-effort race measured the micro candidate at 707 µs against 582 µs for the winning dynamic configuration (cold-cache protocol), so the selection follows the measurement. The draft's 4-row top-3 layers do run the tiny-decode kernel.
  • One dense GEMM (grid (1,1,80)) takes 114 µs per layer in the width-6 round against 21 µs at width one, but 99 % of it overlaps the routed MoE on the other stream, so it does not lengthen the round.
  • _markov_step adds about 1.7 ms per round, and a BF16 GEMV that takes 6 µs at width one takes 188 µs at width six (about 1.8 ms per round).

Question

On this setup one-stream decode is about 45 tok/s on prose and 80 on code, and most of the gap between a width-one and a speculative round is the routed MoE at 4-6 rows. Is a faster small-M routed-MoE path for these verification shapes in scope for b12x? I can collect more data if useful: other batch sizes, --autotune-effort audit, or ncu on the dynamic kernel.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.