128 shared snapshot slots cap multi-user prefix reuse on a 4-Spark DeepSeek TP4 ring

Aperta
  • Qualcosa non va?
#47 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning, performance

Direzione di ricerca

Start by tracing how the inference engine allocates shared snapshot slots and how snapshots are published and restored; the issue does not identify the relevant files or tests. Check whether the 128-slot limit comes from a fixed cap or a tunable package setting, then compare slot behavior across KV pool sizes. Done means establishing whether the count can scale safely and whether snapshot cost grows with message count, with measurements on the reported four-Spark setup.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:caching area:memory area:selene enhancement model:deepseek-v4.1 performance triage:verify-for-closure
Summary

On a four-Spark TP4 ring, DeepSeek V4.1 Flash gets a 25M-token KV pool but only 128 shared snapshot slots, and
the slot count didn't change when we shrank the pool. In a multi-user chat workload, reuse is bounded by the slots,
not by memory. Once other users' prompts publish enough snapshots, a returning user's prefix is fully re-prefilled,
even with most of the pages free. We'd like to know whether the slot count can be raised, or tied to the pool,
on large-pool multi-host packages.

Setup
  • 4× DGX Spark (GB10, SM121), flare-inference 0.1.0~git20261021004534.bb204798, flare orbit run --roce-ring
  • Package local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revision gb10
  • --max-concurrency 16 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB
  • GPU clocks capped at 2100 MHz

Startup, default KV limit:

[rank 0 spark01] [    OK    ] Allocate cache.  21.2 GiB · 25258240 KV tokens · 128 shared snapshot slots · limited by memory  0.9s

With --kv-cache-limit concurrency, the pool shrinks but the slots stay the same:

[rank 0 spark01] [    OK    ] Allocate cache.  14.2 GiB · 16777216 KV tokens · 128 shared snapshot slots · limited by concurrency  0.6s

(For comparison, a single-GPU Qwen3.8-Flash-Next package reported 1925120 KV tokens · 64 shared snapshot slots.)

What we measured

Reuse happens at speaker changes (system→user→assistant), which matches the docs. Each speaker change appears to
use one snapshot.

Retention probe. On a quiet server we primed a ~9K-token conversation of real prose, ran some other traffic,
then re-asked with the conversation extended by one turn:

Traffic between prime and re-ask Prefix reused
nothing / 60 s idle / 300 s idle 100%
1 or 3 other small conversations 100%
one 100K cold read, or four 50K cold reads at once 100%
1 or 2 other prompts with 60 speaker changes each 100%
4 other prompts × 60 speaker changes, or 1 × 240, or 2 × 240 0%

Human-paced author simulation. Each simulated author writes, asks book questions, fetches chapters and edits,
with human think-time between turns; 4 or 8 more authors join every 15 minutes. We used a compact layout to keep
speaker changes few: the book in one system message, and older history folded into one pair. Same simulator, seed
and schedule on both engines. "Reply start" is the median TTFT of ordinary turns.

Authors Reply start, Flare Prefix reused, Flare Decode p50, Flare
4 2.0 s 74% 45 tok/s
8 1.7 s 73% 43 tok/s
12 6.2 s 58% 27 tok/s
16 164 s 18% 6 tok/s

In an earlier run with a chattier layout (resumed sessions of 100+ short messages), every turn at just 4 authors
was a full cold read (140–170 s to first token), while about 94% of KV pages were free.

On the same four Sparks and the same simulator, an SGLang TP4 build of the same model held 88% reuse and a 0.56 s
reply start at 16 authors. It overloaded at 24, against 16 for Flare.

Related observation: fully cached long prompts with many messages

A ~208K-token prompt with ~200 messages, reported fully cached (208,030 of 208,047 tokens), still took about 30 s to
first token per question. A ~192K prompt with ~80 messages took 7.2 s. With few messages, warm TTFT is 0.29 s at
10K and 5.8 s at 169K. Cold prefill also slows with message count, at ~14K tokens:

Messages Cold TTFT Warm TTFT
4 6.4 s 0.4 s
30 10.4 s 0.6 s
120 25.7 s 1.0 s
240 44.0 s 1.7 s

That suggests a per-snapshot cost on publish and restore.

Questions
  1. Is 128 a fixed cap, or derived from something we can tune on a multi-host package? It didn't change with the KV
    limit.
  2. Could the slot count scale with the KV pool on large-pool packages? With 25M tokens, memory isn't the constraint
    for this workload.
  3. Is per-snapshot publish/restore cost expected to grow with message count as above?

Multi-host disk prefix caching just landed in #29 (closing #23) and should soften this for returning users; we haven't tested it yet. Raising the GPU slot count would still help. We're happy to rerun the probes against
a branch.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.