Warm multi-host start re-hashes each peer's cached rank on one thread (7 min for a 280 GB DeepSeek V4.1 TP4 rank)

Aperta
  • Qualcosa non va?
#93 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
performance

Direzione di ricerca

The issue identifies helios/src/runtime_package/protection.rs and package/reader.rs; start by reading both verify_stored_data() paths and how verification_workers() is shared. Run the distribution and runtime_package tests, including corrupted_warm_rank_entry_is_rebuilt_and_republished, then compare the warm-start verification time while confirming every payload is still checked and the first failure in payload order is returned.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:caching area:packaging model:deepseek-v4.1 performance triage:design-review
Summary

On a four-Spark TP4 ring, every warm start of DeepSeek V4.1 Flash spends about 7 minutes in
Distribute rank packages over CX7, although no data moves: each peer re-hashes its cached rank
(about 280 GB) in verify_stored_data() before it serves. We understand the check is on purpose
(corrupted_warm_rank_entry_is_rebuilt_and_republished), and we would like to keep it. The cost
comes from how it runs: one thread reads and hashes one payload after another through 4 MiB
buffered reads, at about 0.6 GB/s, while the hardware can do much more. A small change brings the
whole start from 7–9.5 minutes to 71 s with the same check. Would you accept a PR along these
lines, or would you prefer a different approach?

Setup
  • 4× DGX Spark (GB10, SM121), Flare bce032c0, manual-rank mode (flare run --rank N), --roce-ring
  • Package exported from deepseek-ai/DeepSeek-V4.1-Flash-1017 (format 6); each peer's cached rank is
    about 279 GB in 50 files, including two Engram weight planes of 98.3 GB each

Warm start, peers already hold their ranks:

[    OK    ] Wait for tensor-parallel hosts.  3 peer ranks  0.1s
[    OK    ] Distribute rank packages over CX7.  all rank packages verified  514.0s
...
[    OK    ] Start server.  http://127.0.0.1:8001 · 1048576 tokens/request  0.4s

Every other start phase together takes about 22 s.

What we measured on a peer (GB10, NVMe, idle)
verify_stored_data() during the start ~0.6 GB/s (no fabric traffic, ~700 MB/s disk reads)
SHA-256, one core (openssl speed -evp sha256, 4 MiB blocks) 2.5 GB/s
NVMe, one O_DIRECT reader 9.4 GB/s
NVMe, five O_DIRECT readers 13.2 GB/s
One Engram plane, buffered read + hash (as today) 0.67 GB/s
One Engram plane, O_DIRECT read overlapped with hash (dd iflag=direct bs=16M | sha256sum) 1.6 GB/s

Two things hold it back: payloads are hashed one at a time, and within a payload the read and the
hash take turns through the page cache (read-ahead is 128 KB here).

The change we tested

In helios/src/runtime_package/protection.rs, both payload verify_stored_data() paths:

  1. hash payloads on the package reader's verification workers (verification_workers(), at most 8,
    the same pattern and limit as package/reader.rs), and
  2. hash each payload from aligned 16 MiB blocks read through the payload file's cached direct-I/O
    handle (PayloadFile::direct()) on a reader thread, while the previous block is hashed. A
    filesystem that refuses direct I/O (EINVAL) falls back to the current buffered loop.

Every payload is still hashed, and the first failure in payload order is returned.
corrupted_warm_rank_entry_is_rebuilt_and_republished and the other distribution and
runtime_package tests pass (flare 28, helios 117), and a new unit test checks that direct and
buffered digests agree on unaligned ranges across block boundaries and at the end of a file.

Warm start Distribute Start → serving
bce032c0 425–514 s 7–9.5 min
parallel payloads only 184 s 3 min 23 s
parallel payloads + overlapped direct reads 46.5 s 71 s

With parallel payloads alone, the two 98 GB Engram planes set the time, which is why the
direct reads matter. A side effect we like on GB10: the 280 GB no longer passes through the page
cache, which is GPU memory there, just before the KV pool is sized.

The patch against bce032c0 (one file, protection.rs, plus one line in package/reader.rs to share verification_workers(); the unit test is in the same file) is here:
https://github.com/calvarado2004/deepseek-v41-1017-nvfp4-nvidia-flare/blob/main/patches/0009-hash-stored-payloads-on-the-verification-workers.patch

We have run it on our four-Spark ring since 2026-10-26, including a crash-and-restart test. Happy to
open a PR against main, split it into the two steps, or change the block size, buffer count or
worker limit if you have a preference.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.