GLM-5.3 Flash across two hosts: compiled-forward binding fails on a rank-local guard

Abierto
  • ¿Algo no está bien?
#276 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
python
Área
distributed-systems

Línea de trabajo

Read ignition/components/collectives.py around PreparedVarlenGather.__call__ and ignition/preparation/aot.py around _prepare_ranks, then trace how the GLM prefill forwards are built in ignition/recipes/glm53/mla.py. Reproduce with two hosts and context parallelism enabled, and compare with --no-context-parallel. Done means the cross-host export loads and warms compiled forwards on both ranks without the rank-local guard mismatch.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happens

Exporting glm-5.3-flash with tensor parallelism 2 on two DGX Sparks (GB10, one rank per host) fails right after
forward compilation, in "Load and warm compiled forwards". Rank 0's phase line and rank 1's exception:

[    OK    ] Compile model forwards.  2 compiler workers; per r…  43/43  1628.4s
[  FAILED  ] Load and warm compiled forwards.  6/86  27.9s

ignition.preparation.aot.BindingMismatch: rank-local forward does not match compiled guards: GuardDebugInfo(
result=0,
verbose_code_parts=["L['forward'].generation.decoder.layers[3].attention.windows[0][1].local_rank == 0"],
num_guards_executed=17,
user_stack=None)
  2231.1s rank 1: Load and warm compiled forwards failed in 27.89s

My setup:

  • Rank 0: ignition export <local nvidia/GLM-5.3-Flash-FP4-Local directory at 76348ddb> --ranks 2 --rendezvous 192.168.100.10:29630 --gpus 0 --max-concurrency 8
  • Rank 1: ignition join 192.168.100.10:29630 --rank 1 --ranks 2
  • The hosts are connected by a direct ConnectX-7 link pair; collectives use remote direct memory access over
    Converged Ethernet (RoCE).
  • Tree: main at a3e8778c, which includes #273 and #274. It also carries a local change for #271 (Helios's
    vendored RoCE proxy synced to ABI 14). That change touches only helios/ and docs/; the exception above is
    raised in Ignition's aot.py.

What I think happens (as far as I can read the code)

  • Across hosts, GLM's context-parallel prefill gathers attention records window by window:
    ignition/recipes/glm53/mla.py:382-392 creates a PreparedVarlenGather per window when context.remote is set.
  • Since 4a24d78a ("Let Inductor plan model temporary storage"), PreparedVarlenGather.__call__ selects its own row
    with output[self.local_rank] (ignition/components/collectives.py:888), where self.local_rank is this rank's
    index (:867). Before that commit, __call__ passed self.counts, self.output and the handle to the op and
    didn't read the rank index.
  • _prepare_ranks (ignition/preparation/aot.py:885) compiles each forward once per compilation_variant and
    binds that artifact on every rank (:1074). The GLM prefill forwards keep the default "shared" variant. The
    shared artifact requires local_rank == 0, which appears to explain rank 1's binding failure.
  • Only the separate-hosts path builds these windows. Ranks on one host read each other's caches through CUDA
    interprocess communication (IPC) (pull_context), so I'd expect exports with tensor parallelism on a single host
    not to hit this. I haven't checked that.

Evidence

  • One failed export, 2026-11-12, quoted above. Before it, rank 0 logged every earlier phase OK (weights, collectives,
    graph catalog, kernel tuning in 283.8 s from the cache of an earlier run, warming the serving graphs, graph
    memory), and rank 1 logged the same phases complete.
  • Right after, on the same hosts and tree, I added --no-context-parallel. As far as I can read mla.py, that
    skips the cross-host windows. Rank 0 then logged compilation 43/43 in 1341.1 s, "Load and warm compiled forwards"
    86/86, and publication of both ranks' payloads and the core. Served with flare run (rank 0 and rank 1, same
    tree), that core started in 142 s and answered chat requests: a greedy request repeated gave identical text, and
    four concurrent requests completed.

Not tested

  • Any fix.
  • Other recipes that use PreparedVarlenGather across hosts; I only looked at GLM.

Possible direction (untested)

A per-rank compilation_variant for the forwards that hold cross-host windows may avoid the guard conflict while
keeping the explicit graph-tensor inputs that temporary-storage planning needs.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.