DeepSeek V4.1 Flash TP3 on GB10: a speculative round costs 1.4-2x a width-one round, a fixed draft cost plus about 4 ms per verified row

オープン
  • 何か問題がありますか？
#153 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
rust
領域
machine-learning, performance

調査の方向性

Start by tracing the routed MoE kernel selection and benchmark path for the reported 4-6 row verification shapes; the issue names MoETinyDecodeKernelBackend and SiluMoEDynamicKernelSilu but gives no source files or tests. Compare kernel timings and selection under the stated setup, then determine whether a faster small-M path fits b12x. Done means a measured improvement to speculative round cost on this setup.

索引モデルが issue の本文から書いたものです。

説明

area:kernels area:scheduling model:deepseek-v4.1 performance triage:design-review

Setup

  • Three DGX Spark GB10 hosts, DeepSeek V4.1 Flash TP3, current main (921afd5c). Packages exported with --max-concurrency 4 and --mtp-draft-tokens 5, and a second one with --mtp-draft-tokens 3; served with --scheduler-policy throughput.
  • One stream, streamed chat completions, temperature 0, thinking off, 512 output tokens. Per round, flare_speculative_gpu_nanoseconds_total matches the wall time, and readiness waits are 0.4-2.2 ms, so the round cost is GPU replay.
  • Kernel times come from nsys launch --cuda-graph-trace=node on rank 0 during one 200-token prose request per package (and one with --mtp-draft-tokens 0 served on all ranks).

Round cost

ms per round tokens per round
width 1 (--mtp-draft-tokens 0) 27.5 1
width 4 (3 drafts) prose 42-46, code 46 prose 1.8-2.4, code 3.1-3.7
width 6 (5 drafts) prose 46-50, code 53-55 prose 2.0-2.5, code 3.9-5.0

Where the extra time goes

Routed MoE per layer on rank 0:

1 row 4 rows 6 rows
target layers (40, top-6) 164 µs, MoETinyDecodeKernelBackend (two phases) 419 µs, SiluMoEDynamicKernelSilu 433 µs, SiluMoEDynamicKernelSilu
draft layers (3, top-3) not run 188 µs, MoETinyDecodeKernelBackend 308 µs, SiluMoEDynamicKernelSilu
  • Over the 40 target layers, the routed MoE goes from about 6.6 ms per round at width one to about 17 ms at widths 4 and 6.
  • The dynamic kernel takes almost the same time for 4 and 6 rows, although 6 rows can touch up to 36 experts and 4 rows up to 24.
  • For the 4-row top-6 shape, the default-effort race measured the micro candidate at 707 µs against 582 µs for the winning dynamic configuration (cold-cache protocol), so the selection follows the measurement. The draft's 4-row top-3 layers do run the tiny-decode kernel.
  • One dense GEMM (grid (1,1,80)) takes 114 µs per layer in the width-6 round against 21 µs at width one, but 99 % of it overlaps the routed MoE on the other stream, so it does not lengthen the round.
  • _markov_step adds about 1.7 ms per round, and a BF16 GEMV that takes 6 µs at width one takes 188 µs at width six (about 1.8 ms per round).

Question

On this setup one-stream decode is about 45 tok/s on prose and 80 on code, and most of the gap between a width-one and a speculative round is the routed MoE at 4-6 rows. Is a faster small-M routed-MoE path for these verification shapes in scope for b12x? I can collect more data if useful: other batch sizes, --autotune-effort audit, or ncu on the dynamic kernel.

主要言語
Rust
スター
800
フォーク
106
平均マージ
1日 3時間
マージ済み PR（30日）
100

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/hyperflare-inference のほかの issue

NVIDIA/hyperflare-inference の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。