DeepSeek V4.1 TP4 export on GB10: Xid 13 "Illegal Instruction Encoding" while warming compiled forwards at eb71702b (bce032c works)

オープン
  • 何か問題がありますか？
#137 コメント 10 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
36/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
backend, performance

調査の方向性

Start by comparing the export and warmup paths at bce032c and eb71702b, then trace the kernels used by packed-speculative4x6-drafts5-tokens21 and prefill3072 for SM121. The report names no source files or tests, so first identify how to capture the failing kernel and compare the intervening FlashInfer and kernel changes. Done means finding and reporting the change or configuration that reproduces or resolves the illegal instruction.

索引モデルが issue の本文から書いたものです。

説明

area:preparation bug model:deepseek-v4.1 priority:P2 triage:investigate

Hi, and thank you for Flare. We run DeepSeek-V4.1-Flash-1017 at TP=4 on four DGX Spark (GB10) nodes on the RoCE ring, and it serves well from a bce032c package. When we moved to main at eb71702b, the export fails in "Warm compiled forwards" with a GPU fault on one rank. We are not sure where to look next and would appreciate advice.

What happens

Two runs with the same options. Each one fails on a different node and in a different graph, but the Xid ESR values are identical:

Run Rank (node) Forward being warmed Step
1 2 packed-speculative4x6-drafts5-tokens21 Warm compiled forwards 29/72
2 (cache warm) 3 prefill3072 Warm compiled forwards 2/72 
NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Illegal Instruction Encoding
NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x505730=0x20009 0x505734=0x20 0x505728=0x1c81fb60 0x50572c=0x1174
NVRM: Xid (PCI:000f:01:00): 43, pid=..., name=ignition, channel 0x00000002

The faulting rank reports CUDA_ERROR_ILLEGAL_INSTRUCTION (715). The other ranks then report RoCE collective on rank N timed out waiting for rank 2 (run 1) or RoCE proxy failed: two-shot op 475 abandoned: kernel timed out (run 2). NIC counters show no discards, receive errors or link-down events, so we read the RoCE errors as a consequence of the fault.

Setup

  • 4x DGX Spark GB10 (compute capability 12.1), DGX OS OTA 26.09.2, driver 580.178.04, CUDA 13.3
  • Flare eb71702b with four small local patches (frontend tool choice, image placeholder text, prefix snapshot eviction, payload hashing). None of them touch kernels or Ignition preparation.
  • torch 2.14.1+cu132, triton 3.8.0, flashinfer-python 0.7.1 (local-inference-lab/flashinfer@f91f257e), nvidia-cutlass-dsl 4.7.1
  • ignition export <official checkpoint> --gpus 0 --ranks 4 --roce-ring --transfer-streams 8 --max-concurrency 4 --mtp-draft-tokens 5 --prefill-chunk-size 4096 --expert-scale-compression csf
  • The same options at bce032c export and serve with no fault, on the same OS and hardware, today.

Questions

  1. Is a kernel on this path known to need a newer FlashInfer or b12x build for SM121? We noticed 085ae4a7 (FlashInfer MXFP4 MoE), #109 (MXFP8 attention) and #95 (fused draft scoring) between the two pins, but we have not bisected.
  2. Is there a flag or an environment setting that makes the faulting kernel easier to identify, for example by disabling per-rank tuning selection, so that we can send a more useful report?

We can run a bisect or a debug build on the four nodes if that helps. Thank you.

主要言語
Rust
スター
800
フォーク
106
平均マージ
1日 3時間
マージ済み PR（30日）
100

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/hyperflare-inference のほかの issue

NVIDIA/hyperflare-inference の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。