GB10 TP2 serving: first request times out on RoCE collective (HCA 0, sequence 1) when both CX-7 rails are active; single rail works

Abierto
  • ¿Algo no está bien?
#151 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
distributed-systems, networking

Línea de trabajo

Start by tracing native serving's RoCE transport discovery and collective data path, then compare the multi-rail setup with the single-rail workaround described in the issue. Reproduce the first collective timeout on two active rails and inspect per-rail behavior; done means multi-rail serving completes the first request, or startup reports a clear rail-specific failure.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:rocenante bug model:glm-5.3 priority:P2 triage:investigate

GB10 TP2 serving: first request times out on RoCE collective (HCA 0, sequence 1) when both CX-7 rails are active; single rail works

Summary

On two DGX Sparks (TP2, direct link), flare run starts cleanly and reports 2 rails per peer, but the first chat request kills the run:

RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned

This reproduces on every launch (3/3). The waiting rank varies (rank 0 waiting for rank 1 in two runs, rank 1 waiting for rank 0 in one), but it is always HCA 0 and always sequence 1, so the very first collective never completes in either direction.

When the second CX-7 function (enP2p1s0f1np1 / roceP2p1s0f1) is brought down on both nodes, the native server selects one rail and serving works normally. Fabric-level tests on both rails pass (details below).

Environment

  • 2x DGX Spark (GB10), Ubuntu 24.04.5 LTS, kernel 7.0.0-1019-nvidia, driver 580.178.04, Docker 29.6.2 (same on both nodes)
  • CX-7 firmware 28.50.1002 on all functions, direct cable, MTU 9000
  • Two active functions per node, one subnet each:
    • rocep1s0f1 / enp1s0f1np1: 10.100.32.1 ↔ 10.100.32.2 (RoCE v2 IPv4 GID index 3)
    • roceP2p1s0f1 / enP2p1s0f1np1: 10.100.33.1 ↔ 10.100.33.2 (RoCE v2 IPv4 GID index 3)
    • rocep1s0f0 / roceP2p1s0f0: DOWN
  • rdma-core 50.0 (host and container)
  • Flare at d5fd9824593c2a70f6bdc001b39f16b4c5b3b0b4, single image for Ignition + native Flare
  • Model: nvidia/GLM-5.3-Flash-FP4-Local, TP2 package, --max-concurrency 12 --mtp-draft-tokens 5 --prefill-chunk-size 8192
  • Launched with a third-party multi-host launcher (Docker + SSH, --network host --device /dev/infiniband --ipc host). Serving uses flare run ... --rank N --rendezvous <head>:<port> --gpus 0 with no fabric pinning, so native fabric discovery applies.

Logs

Rank 0 (trimmed):

[    OK    ] Wait for tensor-parallel hosts.  1 peer ranks  1.5s
[    OK    ] Distribute rank packages over CX7.  all rank packages verified  0.0s
[    OK    ] Load weights.  84.1 GiB  15.9s
[    OK    ] Prepare kernels.  3370 kernels  0.7s
[    OK    ] Prepare execution.  concurrency 12  1.2s
[    OK    ] Allocate cache.  9.5 GiB · 2716672 KV tokens · 96 shared snapshot slots · limited by memory  0.6s
[    OK    ] Connect RoCE transport.  2 rails per peer over roceP2p1s0f1,rocep1s0f1  0.2s
[    OK    ] Start server.  http://0.0.0.0:8000 · 1048576 tokens/request  0.6s
[...] request_id=chatcmpl-1-1 stage=generation code=server_error message="tensor-parallel peer failed: RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned"
[  FAILED  ] Shut down server.  0.0s
Error: "native replay did not drain: tensor-parallel peer failed: RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned"

Rank 1:

[    OK    ] Connect RoCE transport.  2 rails per peer over roceP2p1s0f1,rocep1s0f1  0.1s
Serving rank 1 of 2 for rank zero at 10.100.32.1:29800
Error: "native replay did not drain: RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned"

Earlier run, same package (direction reversed):

message="RoCE collective on rank 0 timed out waiting for rank 1, HCA 0, at sequence 1; the transport is poisoned"

What was ruled out

Check Result
Link state / IP / MTU Both rails ACTIVE on both nodes, MTU 9000, jumbo ping -M do -s 8972 OK on both subnets
Host-memory RDMA, ib_write_bw -x 3 per rail roceP2p1s0f1: 109.09 Gb/s, rocep1s0f1: 109.04 Gb/s
NCCL 2.31.2 all_reduce_perf, NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1, NCCL_IB_GID_INDEX=3 Pass, 8 B to 4 GiB, 0 wrong, busbw 23.6 GB/s at 4 GiB; channels alternate NET/IB/0 and NET/IB/1
Kernel log around the failure No mlx5 / RDMA / Xid messages on either node (unlike #137, where the RoCE timeout follows a GPU fault)
Driver / firmware / kernel parity Identical on both nodes

NCCL reported GPU Direct RDMA Disabled / GDR 0. That is expected: the DGX Spark porting guide states that GPUDirect RDMA (nvidia-peermem, dma-buf, GDRCopy) is not supported on this platform and that RDMA buffers should come from cudaHostAlloc + ibv_reg_mr. So the NCCL run used the same kind of host-allocated, NIC-registered buffers on both rails and passed.

Workaround

On both nodes, before launching:

sudo ip link set enP2p1s0f1np1 down

The transport then reports a single rail (rocep1s0f1), and requests complete normally. The second rail's bandwidth is lost.

Notes / requests

  1. Native serving gives no way to choose or exclude rails. flare run --help has no fabric/HCA option, and no FLARE_* environment variable for it was found in the binary. An option such as --roce-hca rocep1s0f1[,…] (or honoring B12X_ROCE_HCA in serving) would allow single-rail operation without taking links down.
  2. Connect RoCE transport succeeds, so QP setup on roceP2p1s0f1 passes, but the first data-path collective on that rail never arrives. A short per-rail data exchange during transport setup (before Start server) would turn this into a startup failure that names the rail, instead of a fatal error on the first user request.
  3. Happy to run diagnostics (debug logging, per-rail counters, a build with extra tracing) if that helps.

Possibly related environment: #143 (same hardware/kernel/driver, two active CX-7 functions, preparation-side GID selection).

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.