Let preparation peers read their own checkpoint copy (multi-host load 252 s -> 45-50 s on GB10)

Aperta
  • Qualcosa non va?
#126 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
distributed-systems

Direzione di ricerca

Start by reading the multi-host preparation and checkpoint admission paths, then trace how rank zero sends the catalog and metadata to peers. The proposed change adds local hub-cache resolution, file identity checks, and require/prefer fallback behavior; done means the option behaves as described and the reported three-host load and inference results can be reproduced.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:preparation model:deepseek-v4.1 performance triage:design-review

Problem

In multi-host preparation, every peer streams all of its checkpoint ranges from rank zero through its loopback relay, even when each host already holds the same Hugging Face snapshot. On three DGX Spark GB10 hosts (DeepSeek V4.1 Flash, TP3), a peer loads at about 430 MB/s that way:

Rank zero (local NVMe) Peers (streamed from rank zero)
Load weights ~46 s 252 s
"Allocate model state and collectives" on rank zero (waits for the peers) 190 s

The peers' own NVMe drives read at 5.0–5.7 GB/s with O_DIRECT dd.

Proposal

Add an opt-in ignition join --local-checkpoint [require|prefer]. Without the flag, behaviour is unchanged.

  • Resolving the checkpoint. The peer looks up rank zero's checkpoint in its own hub cache, with the same repository and revision. For a directory checkpoint, it uses the same absolute path.
  • Admission. Admission still receives rank zero's catalog and metadata over the authenticated channel. A safetensors file is read locally only if its size and header digest match rank zero's. Every other file still comes from rank zero.
  • Modes. require (the default) fails the join on any missing or different tensor file. prefer requests those files from rank zero and reads the rest locally.
  • Reads. Admitted files go to b12x's native batch reader, through O_DIRECT descriptors private to the session, as rank zero reads its own files. The file identity is checked before and after every batch.

Measured (three GB10 hosts, TP3, page cache dropped)

main --local-checkpoint
Load weights on the peers 252 s 44.7–50.1 s
"Allocate model state and collectives" on rank zero 190 s 1.7–5.1 s

A package exported with this path serves correctly with flare run: 262k-token retrieval 16/16, decode at 43.0 tok/s with one stream and 91.8 tok/s aggregate with four.

Limitation

Beyond the header digests, the tensor payload bytes are trusted to be the same revision: the hub cache stores them by content hash. They are not hashed against rank zero's copy.

Branch

javieratomic:local-checkpoint has two commits on top of a161b17f. Because this changes multi-host preparation scope, I would like a maintainer decision before opening a PR.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.