DeepSeek 4.1 TP3 on GB10: 0.46 GiB/GPU less prefill scratch gave +7.6 % KV tokens (measurement only)

Aperta
  • Qualcosa non va?
#186 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
performance

Direzione di ricerca

Start by reading AGENTS.md and the design review’s principle 14, then inspect the DeepSeek V4.1 Flash recipe and the prefill memory layout on main at f571a550. Reproduce the reported TP3 measurement with the listed export and serving commands, comparing several alternating starts because KV capacity varies with free memory. The report supplies measurements but proposes no implementation or acceptance criteria, so further work needs a maintainer decision on whether to pursue the memory tradeoff.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:memory area:preparation model:deepseek-v4.1 performance triage:design-review

On three GB10 hosts at TP3, DeepSeek V4.1 Flash starts with about 7.7M KV tokens (flare_total_kv_tokens). A local prototype that reduced the fixed prefill scratch of the DeepSeek 4.1 recipe by 0.46 GiB per GPU raised that to about 8.3M (+7.6 % on the mean of three starts), with bit-identical first-token log-probabilities. I'm reporting the measurement so you can judge whether the headroom is worth pursuing. I'm not proposing an implementation: the prototype was written with Claude Opus 5.5 and falls under the model-optimization restriction in AGENTS.md and principle 14 of the design review, so I won't submit it or describe it.

Setup. main at f571a550. Both packages were exported with --max-concurrency 4 --mtp-draft-tokens 5 and the default 4096-token prefill chunk, with the preparation host-memory floor lowered from 8 GiB to 6 GiB by a local override (main has no option for it). Two export attempts closer to the recipe defaults (max concurrency 32, 7 draft tokens) failed on these hosts: one at the host-memory check before compilation (12,269 MiB available, 12,288 required), one at a RoCE collective timeout while warming compiled forwards. Both packages were served with flare run from the same main build. Each package was started three times: the first start of each was a cold start without a fixed idle period; the second and third alternated between the packages, each after 3 minutes idle. The prototype's first start attempt failed at "Distribute rank packages over CX7" (failed to fill whole buffer) and started normally on retry.

Decode and prefill in tok/s, aggregate over the streams, mean and range of 3 starts:

main Prototype Difference
Fixed device memory per GPU (flare inspect --memory-map) 96.71 GiB 96.25 GiB -0.46 GiB
flare_total_kv_tokens (3 starts) 7,915,264 / 7,456,512 / 7,762,304 8,505,088 / 8,213,760 / 8,172,672 +7.6 % on the mean
Decode prose x1 45.5 (44.5–46.3) 44.2 (43.7–44.8) -2.9 %
Decode prose x4 89.7 (88.3–91.7) 92.3 (90.3–94.6) +2.9 %
Decode code x1 82.6 (79.4–88.2) 83.1 (80.3–87.0) +0.7 %
Decode code x4 150.4 (142.4–154.9) 149.2 (144.0–157.5) -0.8 %
Prefill, 30k-token prompt 2,764 (2,540–2,891) 2,776 (2,557–2,900) +0.4 %
Prefill, 120k-token prompt 3,390 (3,281–3,448) 3,298 (3,205–3,421) -2.7 %

Every decode and prefill difference is within the range of single starts, but prose x1 and 120k prefill were lower with the prototype in two of the three alternating pairs; three starts cannot separate that from noise. The KV total is sized at start from the memory then free and varied by up to 0.46M tokens between starts of the same package; the lowest prototype start (8.17M) is still above the highest main start (7.92M). At 890 bytes per token per GPU, 0.46 GiB corresponds to about 0.55M tokens; the measured mean difference is 0.59M.

The top-20 log-probabilities of the first generated token were bit-identical between the first start of each package for ten prompts from 836 to 94,341 tokens, and a 118k-token retrieval test passed 16/16 in all six starts.

The saving scales with the prefill chunk: a layout calculation for the prototype gives about 110 MiB per 1024 tokens of chunk, about 440 MiB at 4096, close to the 468 MiB measured. I have not measured other chunk sizes on this base or other recipes.

To reproduce the measurement on main.

  1. Export at TP3, with ignition join <rank0>:<port> --rank {1,2} --ranks 3 --gpu 0 on the peers: 
    ignition export deepseek-v4.1-flash --ranks 3 --rendezvous <rank0>:<port> --gpus 0 \
  --max-concurrency 4 --mtp-draft-tokens 5
    For both packages we lowered ignition.preparation.host_memory._HEADROOM_BYTES to 6 GiB; we did not try this recipe with the default 8 GiB.
  2. Fixed device memory per rank: flare inspect --unverified --memory-map --json <package>/model.safetensors, summing ranks[*].allocations[*].nbytes.
  3. Serve with flare run <package> --rank 0 --rendezvous <rank0>:<port> --gpus 0 --scheduler-policy throughput --prefix-cache-memory 0 --prefix-cache-disk <dir> --prefix-cache-disk-limit 200G (peers: flare run --rank {1,2} ...), after sync; echo 3 > /proc/sys/vm/drop_caches on every host, and read flare_total_kv_tokens from /metrics once /v1/models answers. On GB10 the KV pool is sized from MemAvailable at start, so compare several alternating starts rather than one.
  4. Decode: 512 generated tokens per stream, prose and code prompts, 1 and 4 concurrent streams. Prefill: time to first token for 30k- and 120k-token prompts.

If you look into it, I can export and measure any branch on the same three hosts with the same protocol.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.