GLM-5.3 Flash across two hosts: compiled-forward binding fails on a rank-local guard

オープン
  • 何か問題がありますか？
#276 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
python
領域
distributed-systems

調査の方向性

Read ignition/components/collectives.py around PreparedVarlenGather.__call__ and ignition/preparation/aot.py around _prepare_ranks, then trace how the GLM prefill forwards are built in ignition/recipes/glm53/mla.py. Reproduce with two hosts and context parallelism enabled, and compare with --no-context-parallel. Done means the cross-host export loads and warms compiled forwards on both ranks without the rank-local guard mismatch.

索引モデルが issue の本文から書いたものです。

説明

What happens

Exporting glm-5.3-flash with tensor parallelism 2 on two DGX Sparks (GB10, one rank per host) fails right after
forward compilation, in "Load and warm compiled forwards". Rank 0's phase line and rank 1's exception:

[    OK    ] Compile model forwards.  2 compiler workers; per r…  43/43  1628.4s
[  FAILED  ] Load and warm compiled forwards.  6/86  27.9s

ignition.preparation.aot.BindingMismatch: rank-local forward does not match compiled guards: GuardDebugInfo(
result=0,
verbose_code_parts=["L['forward'].generation.decoder.layers[3].attention.windows[0][1].local_rank == 0"],
num_guards_executed=17,
user_stack=None)
  2231.1s rank 1: Load and warm compiled forwards failed in 27.89s

My setup:

  • Rank 0: ignition export <local nvidia/GLM-5.3-Flash-FP4-Local directory at 76348ddb> --ranks 2 --rendezvous 192.168.100.10:29630 --gpus 0 --max-concurrency 8
  • Rank 1: ignition join 192.168.100.10:29630 --rank 1 --ranks 2
  • The hosts are connected by a direct ConnectX-7 link pair; collectives use remote direct memory access over
    Converged Ethernet (RoCE).
  • Tree: main at a3e8778c, which includes #273 and #274. It also carries a local change for #271 (Helios's
    vendored RoCE proxy synced to ABI 14). That change touches only helios/ and docs/; the exception above is
    raised in Ignition's aot.py.

What I think happens (as far as I can read the code)

  • Across hosts, GLM's context-parallel prefill gathers attention records window by window:
    ignition/recipes/glm53/mla.py:382-392 creates a PreparedVarlenGather per window when context.remote is set.
  • Since 4a24d78a ("Let Inductor plan model temporary storage"), PreparedVarlenGather.__call__ selects its own row
    with output[self.local_rank] (ignition/components/collectives.py:888), where self.local_rank is this rank's
    index (:867). Before that commit, __call__ passed self.counts, self.output and the handle to the op and
    didn't read the rank index.
  • _prepare_ranks (ignition/preparation/aot.py:885) compiles each forward once per compilation_variant and
    binds that artifact on every rank (:1074). The GLM prefill forwards keep the default "shared" variant. The
    shared artifact requires local_rank == 0, which appears to explain rank 1's binding failure.
  • Only the separate-hosts path builds these windows. Ranks on one host read each other's caches through CUDA
    interprocess communication (IPC) (pull_context), so I'd expect exports with tensor parallelism on a single host
    not to hit this. I haven't checked that.

Evidence

  • One failed export, 2026-11-12, quoted above. Before it, rank 0 logged every earlier phase OK (weights, collectives,
    graph catalog, kernel tuning in 283.8 s from the cache of an earlier run, warming the serving graphs, graph
    memory), and rank 1 logged the same phases complete.
  • Right after, on the same hosts and tree, I added --no-context-parallel. As far as I can read mla.py, that
    skips the cross-host windows. Rank 0 then logged compilation 43/43 in 1341.1 s, "Load and warm compiled forwards"
    86/86, and publication of both ranks' payloads and the core. Served with flare run (rank 0 and rank 1, same
    tree), that core started in 142 s and answered chat requests: a greedy request repeated gave identical text, and
    four concurrent requests completed.

Not tested

  • Any fix.
  • Other recipes that use PreparedVarlenGather across hosts; I only looked at GLM.

Possible direction (untested)

A per-rank compilation_variant for the forwards that hold cross-host windows may avoid the guard conflict while
keeping the explicit graph-tensor inputs that temporary-storage planning needs.

主要言語
Rust
スター
800
フォーク
106
平均マージ
1日 3時間
マージ済み PR（30日）
100

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/hyperflare-inference のほかの issue

NVIDIA/hyperflare-inference の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。