DeepSeek V4.1 TP4 export on GB10: Xid 13 "Illegal Instruction Encoding" while warming compiled forwards at eb71702b (bce032c works)

Abierto
  • ¿Algo no está bien?
#137 10 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
36/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, performance

Línea de trabajo

Start by comparing the export and warmup paths at bce032c and eb71702b, then trace the kernels used by packed-speculative4x6-drafts5-tokens21 and prefill3072 for SM121. The report names no source files or tests, so first identify how to capture the failing kernel and compare the intervening FlashInfer and kernel changes. Done means finding and reporting the change or configuration that reproduces or resolves the illegal instruction.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:preparation bug model:deepseek-v4.1 priority:P2 triage:investigate

Hi, and thank you for Flare. We run DeepSeek-V4.1-Flash-1017 at TP=4 on four DGX Spark (GB10) nodes on the RoCE ring, and it serves well from a bce032c package. When we moved to main at eb71702b, the export fails in "Warm compiled forwards" with a GPU fault on one rank. We are not sure where to look next and would appreciate advice.

What happens

Two runs with the same options. Each one fails on a different node and in a different graph, but the Xid ESR values are identical:

Run Rank (node) Forward being warmed Step
1 2 packed-speculative4x6-drafts5-tokens21 Warm compiled forwards 29/72
2 (cache warm) 3 prefill3072 Warm compiled forwards 2/72 
NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Illegal Instruction Encoding
NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x505730=0x20009 0x505734=0x20 0x505728=0x1c81fb60 0x50572c=0x1174
NVRM: Xid (PCI:000f:01:00): 43, pid=..., name=ignition, channel 0x00000002

The faulting rank reports CUDA_ERROR_ILLEGAL_INSTRUCTION (715). The other ranks then report RoCE collective on rank N timed out waiting for rank 2 (run 1) or RoCE proxy failed: two-shot op 475 abandoned: kernel timed out (run 2). NIC counters show no discards, receive errors or link-down events, so we read the RoCE errors as a consequence of the fault.

Setup

  • 4x DGX Spark GB10 (compute capability 12.1), DGX OS OTA 26.09.2, driver 580.178.04, CUDA 13.3
  • Flare eb71702b with four small local patches (frontend tool choice, image placeholder text, prefix snapshot eviction, payload hashing). None of them touch kernels or Ignition preparation.
  • torch 2.14.1+cu132, triton 3.8.0, flashinfer-python 0.7.1 (local-inference-lab/flashinfer@f91f257e), nvidia-cutlass-dsl 4.7.1
  • ignition export <official checkpoint> --gpus 0 --ranks 4 --roce-ring --transfer-streams 8 --max-concurrency 4 --mtp-draft-tokens 5 --prefill-chunk-size 4096 --expert-scale-compression csf
  • The same options at bce032c export and serve with no fault, on the same OS and hardware, today.

Questions

  1. Is a kernel on this path known to need a newer FlashInfer or b12x build for SM121? We noticed 085ae4a7 (FlashInfer MXFP4 MoE), #109 (MXFP8 attention) and #95 (fused draft scoring) between the two pins, but we have not bisected.
  2. Is there a flag or an environment setting that makes the faulting kernel easier to identify, for example by disabling per-rank tuning selection, so that we can send a more useful report?

We can run a bisect or a debug build on the four nodes if that helps. Thank you.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.