Warm multi-host start re-hashes each peer's cached rank on one thread (7 min for a 280 GB DeepSeek V4.1 TP4 rank)

オープン
  • 何か問題がありますか？
#93 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
performance

調査の方向性

The issue identifies helios/src/runtime_package/protection.rs and package/reader.rs; start by reading both verify_stored_data() paths and how verification_workers() is shared. Run the distribution and runtime_package tests, including corrupted_warm_rank_entry_is_rebuilt_and_republished, then compare the warm-start verification time while confirming every payload is still checked and the first failure in payload order is returned.

索引モデルが issue の本文から書いたものです。

説明

area:caching area:packaging model:deepseek-v4.1 performance triage:design-review
Summary

On a four-Spark TP4 ring, every warm start of DeepSeek V4.1 Flash spends about 7 minutes in
Distribute rank packages over CX7, although no data moves: each peer re-hashes its cached rank
(about 280 GB) in verify_stored_data() before it serves. We understand the check is on purpose
(corrupted_warm_rank_entry_is_rebuilt_and_republished), and we would like to keep it. The cost
comes from how it runs: one thread reads and hashes one payload after another through 4 MiB
buffered reads, at about 0.6 GB/s, while the hardware can do much more. A small change brings the
whole start from 7–9.5 minutes to 71 s with the same check. Would you accept a PR along these
lines, or would you prefer a different approach?

Setup
  • 4× DGX Spark (GB10, SM121), Flare bce032c0, manual-rank mode (flare run --rank N), --roce-ring
  • Package exported from deepseek-ai/DeepSeek-V4.1-Flash-1017 (format 6); each peer's cached rank is
    about 279 GB in 50 files, including two Engram weight planes of 98.3 GB each

Warm start, peers already hold their ranks:

[    OK    ] Wait for tensor-parallel hosts.  3 peer ranks  0.1s
[    OK    ] Distribute rank packages over CX7.  all rank packages verified  514.0s
...
[    OK    ] Start server.  http://127.0.0.1:8001 · 1048576 tokens/request  0.4s

Every other start phase together takes about 22 s.

What we measured on a peer (GB10, NVMe, idle)
verify_stored_data() during the start ~0.6 GB/s (no fabric traffic, ~700 MB/s disk reads)
SHA-256, one core (openssl speed -evp sha256, 4 MiB blocks) 2.5 GB/s
NVMe, one O_DIRECT reader 9.4 GB/s
NVMe, five O_DIRECT readers 13.2 GB/s
One Engram plane, buffered read + hash (as today) 0.67 GB/s
One Engram plane, O_DIRECT read overlapped with hash (dd iflag=direct bs=16M | sha256sum) 1.6 GB/s

Two things hold it back: payloads are hashed one at a time, and within a payload the read and the
hash take turns through the page cache (read-ahead is 128 KB here).

The change we tested

In helios/src/runtime_package/protection.rs, both payload verify_stored_data() paths:

  1. hash payloads on the package reader's verification workers (verification_workers(), at most 8,
    the same pattern and limit as package/reader.rs), and
  2. hash each payload from aligned 16 MiB blocks read through the payload file's cached direct-I/O
    handle (PayloadFile::direct()) on a reader thread, while the previous block is hashed. A
    filesystem that refuses direct I/O (EINVAL) falls back to the current buffered loop.

Every payload is still hashed, and the first failure in payload order is returned.
corrupted_warm_rank_entry_is_rebuilt_and_republished and the other distribution and
runtime_package tests pass (flare 28, helios 117), and a new unit test checks that direct and
buffered digests agree on unaligned ranges across block boundaries and at the end of a file.

Warm start Distribute Start → serving
bce032c0 425–514 s 7–9.5 min
parallel payloads only 184 s 3 min 23 s
parallel payloads + overlapped direct reads 46.5 s 71 s

With parallel payloads alone, the two 98 GB Engram planes set the time, which is why the
direct reads matter. A side effect we like on GB10: the 280 GB no longer passes through the page
cache, which is GPU memory there, just before the KV pool is sized.

The patch against bce032c0 (one file, protection.rs, plus one line in package/reader.rs to share verification_workers(); the unit test is in the same file) is here:
https://github.com/calvarado2004/deepseek-v41-1017-nvfp4-nvidia-flare/blob/main/patches/0009-hash-stored-payloads-on-the-verification-workers.patch

We have run it on our four-Spark ring since 2026-10-26, including a crash-and-restart test. Happy to
open a PR against main, split it into the two steps, or change the block size, buffer count or
worker limit if you have a preference.

主要言語
Rust
スター
800
フォーク
106
平均マージ
1日 3時間
マージ済み PR（30日）
100

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/hyperflare-inference のほかの issue

NVIDIA/hyperflare-inference の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。