Multi-host export: when rank zero rejects a peer's checkpoint completion, the reason is never reported

Aperta
  • Qualcosa non va?
#156 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@0z5a ci sta già lavorando.

Dal 4/11/2026.

  • #198 di @0z5a — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, distributed-systems

Direzione di ricerca

Start by tracing SourceServer.run in checkpoint_source.py and FabricSource.join in checkpoint_fabric.py, then follow the disconnect handling in host_ranks.py. Reproduce the failure using the issue's hard-link procedure during a multi-host export. Done means the originating stream error reaches the logs or peer instead of being reduced to a generic disconnect, while changed checkpoints are still rejected.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:preparation bug model:deepseek-v4.1 priority:P2 triage:review

What happened

A DeepSeek V4.1 Flash TP3 export on three GB10 hosts (a161b17f, checkpoint fabric enabled: fabric TCP, 8 streams per peer) loaded weights, compiled, warmed (72/72) and captured serving graphs on every rank, then failed. The only messages were:

  • Rank zero: flare: a tensor-parallel peer disconnected before shutdown
  • Ranks 1 and 2, from source.finish(...) in serve_peer (host_ranks.py): 
    File "ignition/preparation/checkpoint_source.py", line 36, in _exact
    raise ConnectionError("checkpoint source disconnected")
ConnectionError: checkpoint source disconnected

Neither log says why. The cause was found by elimination: during the export, an unrelated hard link to one of rank zero's checkpoint shards was removed. That changed the shard's st_ctime, which is part of the identity rank zero holds for every served file.

Why the reason is lost

Line numbers refer to a161b17f.

  1. The peer's RemoteSource.finish sends finished on its fabric streams one at a time, before the control channel (checkpoint_source.py, lines 448-450). On rank zero, that stream's SourceServer.run (line 260) calls held.validate() on every served file (line 279). HeldFile.validate compares (st_dev, st_ino, st_size, st_mtime_ns, st_ctime_ns) from the held descriptor and from the path (lines 144 and 171), and raises RuntimeError("checkpoint source changed after admission") (line 166).
  2. run stores the exception in self.error and shuts down that fabric stream's socket (line 295) without sending anything to the peer.
  3. The peer's finish() reads EOF and raises ConnectionError("checkpoint source disconnected"). It then reports {"exit": 1, "error": ...} on its control socket (host_ranks.py, line 722) and exits. Rank zero's control-socket SourceServer is still reading that socket as checkpoint frames, so the report is taken as the start of an oversized frame and ends as a disconnect when the peer exits.
  4. On rank zero, _watch_until_closed (host_ranks.py, line 168), which watches the control sockets (line 307), sees the peer's socket close. It prints a tensor-parallel peer disconnected before shutdown (line 179) and calls os._exit(1) (line 182).

The stored changed after admission error is reachable only through FabricSource.join (checkpoint_fabric.py, line 136). Rank zero calls it after joining the control-socket readers (host_ranks.py, lines 354-357), and those fail first, on the peer's disconnect. So even without the watcher, rank zero would report the wrong cause; in the observed run, the watcher exited the process first.

The rejection itself is correct: a file that changed after admission must not be accepted. The problem is that the reason disappears. The same applies to every error raised in SourceServer.run after admission: the read-time validate() (lines 233 and 237), the completion checks (lines 276-277), and any other failure of a served stream.

Reproduction

During a multi-host export with checkpoint_fabric configured, at any point after "Admit checkpoint range readers" and before the peers' finish(), change the st_ctime of one of rank zero's checkpoint files without changing its contents. For example, create and remove a hard link to it on the same file system:

ln <hub>/blobs/<checkpoint blob> <same filesystem>/x && rm <same filesystem>/x

A change made after loading fails at the end of preparation, with the two messages above. A change made during loading fails a read instead, and its reason is lost the same way. Neither case shows the cause in any log.

Possible directions

  • Before shutting a stream down, send the peer an error message carrying the reason (for example checkpoint source changed after admission: <path>), so the peer raises it instead of a bare disconnect.
  • Or, on rank zero, report the first stored SourceServer.error (fabric or control) before acting on a peer disconnect, and join the fabric readers before the control readers.

Including the file name in the changed after admission message would also help.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.