GB10 TP2 serving: first request times out on RoCE collective (HCA 0, sequence 1) when both CX-7 rails are active; single rail works

Đang mở
  • Có gì không ổn?
#151 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust
Lĩnh vực
distributed-systems, networking

Hướng nghiên cứu

Start by tracing native serving's RoCE transport discovery and collective data path, then compare the multi-rail setup with the single-rail workaround described in the issue. Reproduce the first collective timeout on two active rails and inspect per-rail behavior; done means multi-rail serving completes the first request, or startup reports a clear rail-specific failure.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area:rocenante bug model:glm-5.3 priority:P2 triage:investigate

GB10 TP2 serving: first request times out on RoCE collective (HCA 0, sequence 1) when both CX-7 rails are active; single rail works

Summary

On two DGX Sparks (TP2, direct link), flare run starts cleanly and reports 2 rails per peer, but the first chat request kills the run:

RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned

This reproduces on every launch (3/3). The waiting rank varies (rank 0 waiting for rank 1 in two runs, rank 1 waiting for rank 0 in one), but it is always HCA 0 and always sequence 1, so the very first collective never completes in either direction.

When the second CX-7 function (enP2p1s0f1np1 / roceP2p1s0f1) is brought down on both nodes, the native server selects one rail and serving works normally. Fabric-level tests on both rails pass (details below).

Environment

  • 2x DGX Spark (GB10), Ubuntu 24.04.5 LTS, kernel 7.0.0-1019-nvidia, driver 580.178.04, Docker 29.6.2 (same on both nodes)
  • CX-7 firmware 28.50.1002 on all functions, direct cable, MTU 9000
  • Two active functions per node, one subnet each:
    • rocep1s0f1 / enp1s0f1np1: 10.100.32.1 ↔ 10.100.32.2 (RoCE v2 IPv4 GID index 3)
    • roceP2p1s0f1 / enP2p1s0f1np1: 10.100.33.1 ↔ 10.100.33.2 (RoCE v2 IPv4 GID index 3)
    • rocep1s0f0 / roceP2p1s0f0: DOWN
  • rdma-core 50.0 (host and container)
  • Flare at d5fd9824593c2a70f6bdc001b39f16b4c5b3b0b4, single image for Ignition + native Flare
  • Model: nvidia/GLM-5.3-Flash-FP4-Local, TP2 package, --max-concurrency 12 --mtp-draft-tokens 5 --prefill-chunk-size 8192
  • Launched with a third-party multi-host launcher (Docker + SSH, --network host --device /dev/infiniband --ipc host). Serving uses flare run ... --rank N --rendezvous <head>:<port> --gpus 0 with no fabric pinning, so native fabric discovery applies.

Logs

Rank 0 (trimmed):

[    OK    ] Wait for tensor-parallel hosts.  1 peer ranks  1.5s
[    OK    ] Distribute rank packages over CX7.  all rank packages verified  0.0s
[    OK    ] Load weights.  84.1 GiB  15.9s
[    OK    ] Prepare kernels.  3370 kernels  0.7s
[    OK    ] Prepare execution.  concurrency 12  1.2s
[    OK    ] Allocate cache.  9.5 GiB · 2716672 KV tokens · 96 shared snapshot slots · limited by memory  0.6s
[    OK    ] Connect RoCE transport.  2 rails per peer over roceP2p1s0f1,rocep1s0f1  0.2s
[    OK    ] Start server.  http://0.0.0.0:8000 · 1048576 tokens/request  0.6s
[...] request_id=chatcmpl-1-1 stage=generation code=server_error message="tensor-parallel peer failed: RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned"
[  FAILED  ] Shut down server.  0.0s
Error: "native replay did not drain: tensor-parallel peer failed: RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned"

Rank 1:

[    OK    ] Connect RoCE transport.  2 rails per peer over roceP2p1s0f1,rocep1s0f1  0.1s
Serving rank 1 of 2 for rank zero at 10.100.32.1:29800
Error: "native replay did not drain: RoCE collective on rank 1 timed out waiting for rank 0, HCA 0, at sequence 1; the transport is poisoned"

Earlier run, same package (direction reversed):

message="RoCE collective on rank 0 timed out waiting for rank 1, HCA 0, at sequence 1; the transport is poisoned"

What was ruled out

Check Result
Link state / IP / MTU Both rails ACTIVE on both nodes, MTU 9000, jumbo ping -M do -s 8972 OK on both subnets
Host-memory RDMA, ib_write_bw -x 3 per rail roceP2p1s0f1: 109.09 Gb/s, rocep1s0f1: 109.04 Gb/s
NCCL 2.31.2 all_reduce_perf, NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1, NCCL_IB_GID_INDEX=3 Pass, 8 B to 4 GiB, 0 wrong, busbw 23.6 GB/s at 4 GiB; channels alternate NET/IB/0 and NET/IB/1
Kernel log around the failure No mlx5 / RDMA / Xid messages on either node (unlike #137, where the RoCE timeout follows a GPU fault)
Driver / firmware / kernel parity Identical on both nodes

NCCL reported GPU Direct RDMA Disabled / GDR 0. That is expected: the DGX Spark porting guide states that GPUDirect RDMA (nvidia-peermem, dma-buf, GDRCopy) is not supported on this platform and that RDMA buffers should come from cudaHostAlloc + ibv_reg_mr. So the NCCL run used the same kind of host-allocated, NIC-registered buffers on both rails and passed.

Workaround

On both nodes, before launching:

sudo ip link set enP2p1s0f1np1 down

The transport then reports a single rail (rocep1s0f1), and requests complete normally. The second rail's bandwidth is lost.

Notes / requests

  1. Native serving gives no way to choose or exclude rails. flare run --help has no fabric/HCA option, and no FLARE_* environment variable for it was found in the binary. An option such as --roce-hca rocep1s0f1[,…] (or honoring B12X_ROCE_HCA in serving) would allow single-rail operation without taking links down.
  2. Connect RoCE transport succeeds, so QP setup on roceP2p1s0f1 passes, but the first data-path collective on that rail never arrives. A short per-rail data exchange during transport setup (before Start server) would turn this into a startup failure that names the rail, instead of a fatal error on the first user request.
  3. Happy to run diagnostics (debug logging, per-rail counters, a build with extra tracing) if that helps.

Possibly related environment: #143 (same hardware/kernel/driver, two active CX-7 functions, preparation-side GID selection).

Ngôn ngữ chính
Rust
Star
800
Fork
106
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
100

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/hyperflare-inference

Tất cả issue của NVIDIA/hyperflare-inference

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.