DeepSeek V4.1 TP4 export on GB10: Xid 13 "Illegal Instruction Encoding" while warming compiled forwards at eb71702b (bce032c works)

Aperta
  • Qualcosa non va?
#137 10 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
36/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, performance

Direzione di ricerca

Start by comparing the export and warmup paths at bce032c and eb71702b, then trace the kernels used by packed-speculative4x6-drafts5-tokens21 and prefill3072 for SM121. The report names no source files or tests, so first identify how to capture the failing kernel and compare the intervening FlashInfer and kernel changes. Done means finding and reporting the change or configuration that reproduces or resolves the illegal instruction.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:preparation bug model:deepseek-v4.1 priority:P2 triage:investigate

Hi, and thank you for Flare. We run DeepSeek-V4.1-Flash-1017 at TP=4 on four DGX Spark (GB10) nodes on the RoCE ring, and it serves well from a bce032c package. When we moved to main at eb71702b, the export fails in "Warm compiled forwards" with a GPU fault on one rank. We are not sure where to look next and would appreciate advice.

What happens

Two runs with the same options. Each one fails on a different node and in a different graph, but the Xid ESR values are identical:

Run Rank (node) Forward being warmed Step
1 2 packed-speculative4x6-drafts5-tokens21 Warm compiled forwards 29/72
2 (cache warm) 3 prefill3072 Warm compiled forwards 2/72 
NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Illegal Instruction Encoding
NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x505730=0x20009 0x505734=0x20 0x505728=0x1c81fb60 0x50572c=0x1174
NVRM: Xid (PCI:000f:01:00): 43, pid=..., name=ignition, channel 0x00000002

The faulting rank reports CUDA_ERROR_ILLEGAL_INSTRUCTION (715). The other ranks then report RoCE collective on rank N timed out waiting for rank 2 (run 1) or RoCE proxy failed: two-shot op 475 abandoned: kernel timed out (run 2). NIC counters show no discards, receive errors or link-down events, so we read the RoCE errors as a consequence of the fault.

Setup

  • 4x DGX Spark GB10 (compute capability 12.1), DGX OS OTA 26.09.2, driver 580.178.04, CUDA 13.3
  • Flare eb71702b with four small local patches (frontend tool choice, image placeholder text, prefix snapshot eviction, payload hashing). None of them touch kernels or Ignition preparation.
  • torch 2.14.1+cu132, triton 3.8.0, flashinfer-python 0.7.1 (local-inference-lab/flashinfer@f91f257e), nvidia-cutlass-dsl 4.7.1
  • ignition export <official checkpoint> --gpus 0 --ranks 4 --roce-ring --transfer-streams 8 --max-concurrency 4 --mtp-draft-tokens 5 --prefill-chunk-size 4096 --expert-scale-compression csf
  • The same options at bce032c export and serve with no fault, on the same OS and hardware, today.

Questions

  1. Is a kernel on this path known to need a newer FlashInfer or b12x build for SM121? We noticed 085ae4a7 (FlashInfer MXFP4 MoE), #109 (MXFP8 attention) and #95 (fused draft scoring) between the two pins, but we have not bisected.
  2. Is there a flag or an environment setting that makes the faulting kernel easier to identify, for example by disabling per-rank tuning selection, so that we can send a more useful report?

We can run a bisect or a debug build on the four nodes if that helps. Thank you.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.