GLM-5.3 Flash across two hosts: compiled-forward binding fails on a rank-local guard

Aperta
  • Qualcosa non va?
#276 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python
Ambito
distributed-systems

Direzione di ricerca

Read ignition/components/collectives.py around PreparedVarlenGather.__call__ and ignition/preparation/aot.py around _prepare_ranks, then trace how the GLM prefill forwards are built in ignition/recipes/glm53/mla.py. Reproduce with two hosts and context parallelism enabled, and compare with --no-context-parallel. Done means the cross-host export loads and warms compiled forwards on both ranks without the rank-local guard mismatch.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happens

Exporting glm-5.3-flash with tensor parallelism 2 on two DGX Sparks (GB10, one rank per host) fails right after
forward compilation, in "Load and warm compiled forwards". Rank 0's phase line and rank 1's exception:

[    OK    ] Compile model forwards.  2 compiler workers; per r…  43/43  1628.4s
[  FAILED  ] Load and warm compiled forwards.  6/86  27.9s

ignition.preparation.aot.BindingMismatch: rank-local forward does not match compiled guards: GuardDebugInfo(
result=0,
verbose_code_parts=["L['forward'].generation.decoder.layers[3].attention.windows[0][1].local_rank == 0"],
num_guards_executed=17,
user_stack=None)
  2231.1s rank 1: Load and warm compiled forwards failed in 27.89s

My setup:

  • Rank 0: ignition export <local nvidia/GLM-5.3-Flash-FP4-Local directory at 76348ddb> --ranks 2 --rendezvous 192.168.100.10:29630 --gpus 0 --max-concurrency 8
  • Rank 1: ignition join 192.168.100.10:29630 --rank 1 --ranks 2
  • The hosts are connected by a direct ConnectX-7 link pair; collectives use remote direct memory access over
    Converged Ethernet (RoCE).
  • Tree: main at a3e8778c, which includes #273 and #274. It also carries a local change for #271 (Helios's
    vendored RoCE proxy synced to ABI 14). That change touches only helios/ and docs/; the exception above is
    raised in Ignition's aot.py.

What I think happens (as far as I can read the code)

  • Across hosts, GLM's context-parallel prefill gathers attention records window by window:
    ignition/recipes/glm53/mla.py:382-392 creates a PreparedVarlenGather per window when context.remote is set.
  • Since 4a24d78a ("Let Inductor plan model temporary storage"), PreparedVarlenGather.__call__ selects its own row
    with output[self.local_rank] (ignition/components/collectives.py:888), where self.local_rank is this rank's
    index (:867). Before that commit, __call__ passed self.counts, self.output and the handle to the op and
    didn't read the rank index.
  • _prepare_ranks (ignition/preparation/aot.py:885) compiles each forward once per compilation_variant and
    binds that artifact on every rank (:1074). The GLM prefill forwards keep the default "shared" variant. The
    shared artifact requires local_rank == 0, which appears to explain rank 1's binding failure.
  • Only the separate-hosts path builds these windows. Ranks on one host read each other's caches through CUDA
    interprocess communication (IPC) (pull_context), so I'd expect exports with tensor parallelism on a single host
    not to hit this. I haven't checked that.

Evidence

  • One failed export, 2026-11-12, quoted above. Before it, rank 0 logged every earlier phase OK (weights, collectives,
    graph catalog, kernel tuning in 283.8 s from the cache of an earlier run, warming the serving graphs, graph
    memory), and rank 1 logged the same phases complete.
  • Right after, on the same hosts and tree, I added --no-context-parallel. As far as I can read mla.py, that
    skips the cross-host windows. Rank 0 then logged compilation 43/43 in 1341.1 s, "Load and warm compiled forwards"
    86/86, and publication of both ranks' payloads and the core. Served with flare run (rank 0 and rank 1, same
    tree), that core started in 142 s and answered chat requests: a greedy request repeated gave identical text, and
    four concurrent requests completed.

Not tested

  • Any fix.
  • Other recipes that use PreparedVarlenGather across hosts; I only looked at GLM.

Possible direction (untested)

A per-rank compilation_variant for the forwards that hold cross-host windows may avoid the guard conflict while
keeping the explicit graph-tensor inputs that temporary-storage planning needs.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.