128 shared snapshot slots cap multi-user prefix reuse on a 4-Spark DeepSeek TP4 ring

Abierto
  • ¿Algo no está bien?
#47 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
machine-learning, performance

Línea de trabajo

Start by tracing how the inference engine allocates shared snapshot slots and how snapshots are published and restored; the issue does not identify the relevant files or tests. Check whether the 128-slot limit comes from a fixed cap or a tunable package setting, then compare slot behavior across KV pool sizes. Done means establishing whether the count can scale safely and whether snapshot cost grows with message count, with measurements on the reported four-Spark setup.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:caching area:memory area:selene enhancement model:deepseek-v4.1 performance triage:verify-for-closure
Summary

On a four-Spark TP4 ring, DeepSeek V4.1 Flash gets a 25M-token KV pool but only 128 shared snapshot slots, and
the slot count didn't change when we shrank the pool. In a multi-user chat workload, reuse is bounded by the slots,
not by memory. Once other users' prompts publish enough snapshots, a returning user's prefix is fully re-prefilled,
even with most of the pages free. We'd like to know whether the slot count can be raised, or tied to the pool,
on large-pool multi-host packages.

Setup
  • 4× DGX Spark (GB10, SM121), flare-inference 0.1.0~git20261021004534.bb204798, flare orbit run --roce-ring
  • Package local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revision gb10
  • --max-concurrency 16 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB
  • GPU clocks capped at 2100 MHz

Startup, default KV limit:

[rank 0 spark01] [    OK    ] Allocate cache.  21.2 GiB · 25258240 KV tokens · 128 shared snapshot slots · limited by memory  0.9s

With --kv-cache-limit concurrency, the pool shrinks but the slots stay the same:

[rank 0 spark01] [    OK    ] Allocate cache.  14.2 GiB · 16777216 KV tokens · 128 shared snapshot slots · limited by concurrency  0.6s

(For comparison, a single-GPU Qwen3.8-Flash-Next package reported 1925120 KV tokens · 64 shared snapshot slots.)

What we measured

Reuse happens at speaker changes (system→user→assistant), which matches the docs. Each speaker change appears to
use one snapshot.

Retention probe. On a quiet server we primed a ~9K-token conversation of real prose, ran some other traffic,
then re-asked with the conversation extended by one turn:

Traffic between prime and re-ask Prefix reused
nothing / 60 s idle / 300 s idle 100%
1 or 3 other small conversations 100%
one 100K cold read, or four 50K cold reads at once 100%
1 or 2 other prompts with 60 speaker changes each 100%
4 other prompts × 60 speaker changes, or 1 × 240, or 2 × 240 0%

Human-paced author simulation. Each simulated author writes, asks book questions, fetches chapters and edits,
with human think-time between turns; 4 or 8 more authors join every 15 minutes. We used a compact layout to keep
speaker changes few: the book in one system message, and older history folded into one pair. Same simulator, seed
and schedule on both engines. "Reply start" is the median TTFT of ordinary turns.

Authors Reply start, Flare Prefix reused, Flare Decode p50, Flare
4 2.0 s 74% 45 tok/s
8 1.7 s 73% 43 tok/s
12 6.2 s 58% 27 tok/s
16 164 s 18% 6 tok/s

In an earlier run with a chattier layout (resumed sessions of 100+ short messages), every turn at just 4 authors
was a full cold read (140–170 s to first token), while about 94% of KV pages were free.

On the same four Sparks and the same simulator, an SGLang TP4 build of the same model held 88% reuse and a 0.56 s
reply start at 16 authors. It overloaded at 24, against 16 for Flare.

Related observation: fully cached long prompts with many messages

A ~208K-token prompt with ~200 messages, reported fully cached (208,030 of 208,047 tokens), still took about 30 s to
first token per question. A ~192K prompt with ~80 messages took 7.2 s. With few messages, warm TTFT is 0.29 s at
10K and 5.8 s at 169K. Cold prefill also slows with message count, at ~14K tokens:

Messages Cold TTFT Warm TTFT
4 6.4 s 0.4 s
30 10.4 s 0.6 s
120 25.7 s 1.0 s
240 44.0 s 1.7 s

That suggests a per-snapshot cost on publish and restore.

Questions
  1. Is 128 a fixed cap, or derived from something we can tune on a multi-host package? It didn't change with the KV
    limit.
  2. Could the slot count scale with the KV pool on large-pool packages? With 25M tokens, memory isn't the constraint
    for this workload.
  3. Is per-snapshot publish/restore cost expected to grow with message count as above?

Multi-host disk prefix caching just landed in #29 (closing #23) and should soften this for returning users; we haven't tested it yet. Raising the GPU slot count would still help. We're happy to rerun the probes against
a branch.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.