Multi-host RoCE cores exported at 0572fd12 do not start: pinned FlashInfer 7b35510b records proxy ABI 14, the runtime provides 11

オープン
  • 何か問題がありますか？
#271 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
c, python, rust
領域
distributed-systems, networking

調査の方向性

Start with the three ABI constants the report names: helios/native/roce_proxy.c:103, PROXY_ABI at helios/src/execution/roce.rs:23, and the accepted set in open at roce.rs:125. Compare them against the pinned FlashInfer _roce_proxy.c at flashinfer/experimental/b12x/comm/roce/_roce_proxy.c, then extend native_proxy_abi_matches_core_contract (roce.rs:387-390) to compare the vendored proxy with the pinned library's ABI rather than only with PROXY_ABI. Done means a core recording ABI 14 opens on the rebuilt runtime and the new test fails when the pin and vendored proxy diverge. Note that #232 also changes roce.rs but not these constants.

索引モデルが issue の本文から書いたものです。

説明

What happens

A DeepSeek V4.1 Flash core exported with tensor parallelism 3 on three GB10 hosts (one rank per host, RoCE) does not start under the runtime built from the same tree. Every rank loads its weights and allocates its cache, then stops at "Connect RoCE transport":

[  FAILED  ] Connect RoCE transport.  0.0s
Error: Execution(Poisoned(Failure { message: "core RoCE kernels use proxy ABI 14, this runtime provides 11", source: None }))

The exporter and the runtime of 0572fd12 disagree on the RoCE proxy ABI:

  • c74b613d pins FlashInfer 7b35510b, whose flashinfer/experimental/b12x/comm/roce/_roce_proxy.c defines ROCE_ABI_VERSION 14. local-inference-lab/flashinfer c9b42612 ("Route RoCEnante over direct cables with queue-pair lanes, a partner relay and overrun credits", local-inference-lab/flashinfer#81) raised it from 11. The exporter stores the value that library reports as the proxy_abi of the core's transport (flashinfer/experimental/b12x/comm/roce/roce_oneshot.py:1206).
  • The runtime's vendored proxy still defines ROCE_ABI_VERSION 11 (helios/native/roce_proxy.c:103), PROXY_ABI is 11 (helios/src/execution/roce.rs:23), and open accepts only cores that record ABI 9, 10 or 11 (roce.rs:125).

#251 (e774a468) matched the runtime to ABI 11, which the pin of that time, 85ea6e76, and the later 4e977a53 (d095e9e4) both define. c74b613d replaced ead6cfc1 (b3bf45f9), a revision GitHub no longer resolves, so I could not check its ABI. The message of c74b613d says the pinned revision sits "on top of FlashInfer main's RoCE transport changes" and that "cross-host transport has not been retested here".

Why it matters

A multi-host RoCE core exported at 0572fd12 cannot be served by the runtime of 0572fd12. The mismatch is reported only when a rank connects its transport, after the export has completed (9527 s here) and each start has loaded the weights.

Evidence

  • Five starts of one core, three ranks each: 13 of the 15 logs contain the message above. The other two, one of rank 0 and one of rank 1, end at the failed "Connect RoCE transport" step (rank 0 also prints a failed "Load tokenizer" line) without the error text. The block above is from a peer rank.
  • The core was exported from f25a0386 (#122 on 0572fd12) with --compile-headroom 4G, because the default export of 0572fd12 stops in kernel tuning on these hosts (#270). f25a0386 changes no Rust or C source and no dependency pin relative to 0572fd12.
  • The runtime was built from the same tree with cargo build --release --locked --manifest-path flare-cli/Cargo.toml --features roce.
  • direct_url.json of the installed flashinfer-python 0.7.3 records commit 7b35510b, as uv.lock pins.

Not measured

  • Whether ABI 12 to 14 change what the captured kernels see, or only the proxy's own side. The vendored proxy and the pinned FlashInfer's _roce_proxy.c differ in more than the version constant.
  • Single-host cores and other models. The check runs only where a core records a RoCE transport.

Possible direction

  • The repair #251 made for ABI 11: bring helios/native/roce_proxy.c, PROXY_ABI and the accepted set in open up to the pinned FlashInfer's ABI 14. #232 (open, "Route multi-host RoCE rails over direct cables") changes roce.rs and does not change these constants.
  • A test that compares the vendored proxy's ROCE_ABI_VERSION with the one in the pinned FlashInfer, so that a pin that changes the ABI fails in CI. native_proxy_abi_matches_core_contract (roce.rs:387-390) compares the vendored proxy only with PROXY_ABI.

Environment

  • Three GB10 hosts (DGX Spark class), one rank per host, RoCE between them.
  • Checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017.
  • 0572fd12 runtime; core exported from f25a0386; FlashInfer 7b35510b.
主要言語
Rust
スター
800
フォーク
106
平均マージ
1日 3時間
マージ済み PR（30日）
100

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/hyperflare-inference のほかの issue

NVIDIA/hyperflare-inference の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。