GB10 TP2 export: b12x RoCE prep hard-defaults to GID index 3 and fails when the IPv4 RoCE v2 GID is elsewhere (Helios already scans)

Aperta
  • Qualcosa non va?
#143 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
linux, python
Ambito
backend, networking

Direzione di ricerca

Start with flashinfer/experimental/b12x/comm/roce/roce_oneshot.py and trace how DEFAULT_GID_INDEX, default_gid_index() and discover_hcas() select GIDs; compare the fallback behavior described for helios/src/execution/roce_fabric.rs. Check how B12X_ROCE_GID_INDEX and NCCL_IB_GID_INDEX are handled and look for related tests. Done means preparation selects a RoCE v2 IPv4-mapped GID when the preferred index is unsuitable, while honoring explicit configuration.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:preparation area:rocenante bug model:qwen3.8-flash-next priority:P2 triage:investigate

Summary

On two DGX Sparks (GB10, TP2 over ConnectX-7 RoCE v2), the b12x RoCE preparation step used during export picks GID index 3 unconditionally (DEFAULT_GID_INDEX = 3 in flashinfer/experimental/b12x/comm/roce/roce_oneshot.py, both in local-inference-lab/flashinfer and upstream flashinfer-ai/flashinfer). After a reboot, the GID table on one node was reordered, index 3 held an IPv6 GID, and prep failed with no active RDMA device found. Helios already handles this case correctly: helios/src/execution/roce_fabric.rs prefers index 3 but falls back to the first index that is a RoCE v2 IPv4-mapped GID. The b12x prep has no such fallback. (Filed here because issues are disabled on the LIL flashinfer fork, and Flare's export is where it bites.)

Environment

  • 2x DGX Spark (GB10), Ubuntu 24.04.5, kernel 7.0.0-1019-nvidia, driver 580.178.04, Docker 29.6.2
  • Two active CX-7 ports per node (rocep1s0f0, roceP2p1s0f0), RoCE v2, direct link
  • Flare main d5fd9824 (2026-10-25/26 test), package q38-tp2-c8-mtp, checkpoint nvidia/Qwen3.8-Flash-Next-FP4-Local @b9ce18a6
  • DEFAULT_GID_INDEX = 3 is still present in both flashinfer repos as of 2026-10-29

GID tables (sysfs, IPv4-mapped entries only)

Node Device RoCE v1 RoCE v2
node 0 both ports idx 4 idx 5
node 1 both ports idx 5 idx 6

Index 3 is an IPv6 GID on both nodes. The order is not stable: node 1 moved from 5 to 6 across a reboot, which also broke an unrelated vLLM launch script that hard-coded NCCL_IB_GID_INDEX=3.

Workaround

Read the IPv4 RoCE v2 index from sysfs at launch and pass it in:

for i in $(seq 0 15); do
  t=$(cat /sys/class/infiniband/rocep1s0f0/ports/1/gid_attrs/types/$i 2>/dev/null)
  g=$(cat /sys/class/infiniband/rocep1s0f0/ports/1/gids/$i 2>/dev/null)
  [ "$t" = "RoCE v2" ] && case $g in 0000:0000:0000:0000:0000:ffff:*) echo $i; break;; esac
done

and export it as B12X_ROCE_GID_INDEX to the preparation containers. With that, export completed on both nodes.

Suggested fix

Make default_gid_index() / discover_hcas() follow the Helios rule: use the configured or preferred index only if it is a RoCE v2 IPv4 GID on every selected device, else scan for the first index that is. An explicit B12X_ROCE_GID_INDEX or NCCL_IB_GID_INDEX should still win.

Other observations from the same run (d5fd9824, not yet retested on current main)

  • One export hang in "Compile model forwards": both ranks asleep, GPUs idle, no log output for 45 min. After docker rm -f the retry passed (tuning 48 s, forwards 117 s from cache). Seen once; logs kept if useful.
  • Rank-1 checkpoint transfer over fabric TCP ran at 0.37 Gbit/s (37.5 GiB in 877 s) on the 200G link. 38c59ff8 (Nagle off on fabric relay sockets) and #126 look related; details in a comment there.
Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.