128 shared snapshot slots cap multi-user prefix reuse on a 4-Spark DeepSeek TP4 ring

Đang mở
  • Có gì không ổn?
#47 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
machine-learning, performance

Hướng nghiên cứu

Start by tracing how the inference engine allocates shared snapshot slots and how snapshots are published and restored; the issue does not identify the relevant files or tests. Check whether the 128-slot limit comes from a fixed cap or a tunable package setting, then compare slot behavior across KV pool sizes. Done means establishing whether the count can scale safely and whether snapshot cost grows with message count, with measurements on the reported four-Spark setup.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area:caching area:memory area:selene enhancement model:deepseek-v4.1 performance triage:verify-for-closure
Summary

On a four-Spark TP4 ring, DeepSeek V4.1 Flash gets a 25M-token KV pool but only 128 shared snapshot slots, and
the slot count didn't change when we shrank the pool. In a multi-user chat workload, reuse is bounded by the slots,
not by memory. Once other users' prompts publish enough snapshots, a returning user's prefix is fully re-prefilled,
even with most of the pages free. We'd like to know whether the slot count can be raised, or tied to the pool,
on large-pool multi-host packages.

Setup
  • 4× DGX Spark (GB10, SM121), flare-inference 0.1.0~git20261021004534.bb204798, flare orbit run --roce-ring
  • Package local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revision gb10
  • --max-concurrency 16 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB
  • GPU clocks capped at 2100 MHz

Startup, default KV limit:

[rank 0 spark01] [    OK    ] Allocate cache.  21.2 GiB · 25258240 KV tokens · 128 shared snapshot slots · limited by memory  0.9s

With --kv-cache-limit concurrency, the pool shrinks but the slots stay the same:

[rank 0 spark01] [    OK    ] Allocate cache.  14.2 GiB · 16777216 KV tokens · 128 shared snapshot slots · limited by concurrency  0.6s

(For comparison, a single-GPU Qwen3.8-Flash-Next package reported 1925120 KV tokens · 64 shared snapshot slots.)

What we measured

Reuse happens at speaker changes (system→user→assistant), which matches the docs. Each speaker change appears to
use one snapshot.

Retention probe. On a quiet server we primed a ~9K-token conversation of real prose, ran some other traffic,
then re-asked with the conversation extended by one turn:

Traffic between prime and re-ask Prefix reused
nothing / 60 s idle / 300 s idle 100%
1 or 3 other small conversations 100%
one 100K cold read, or four 50K cold reads at once 100%
1 or 2 other prompts with 60 speaker changes each 100%
4 other prompts × 60 speaker changes, or 1 × 240, or 2 × 240 0%

Human-paced author simulation. Each simulated author writes, asks book questions, fetches chapters and edits,
with human think-time between turns; 4 or 8 more authors join every 15 minutes. We used a compact layout to keep
speaker changes few: the book in one system message, and older history folded into one pair. Same simulator, seed
and schedule on both engines. "Reply start" is the median TTFT of ordinary turns.

Authors Reply start, Flare Prefix reused, Flare Decode p50, Flare
4 2.0 s 74% 45 tok/s
8 1.7 s 73% 43 tok/s
12 6.2 s 58% 27 tok/s
16 164 s 18% 6 tok/s

In an earlier run with a chattier layout (resumed sessions of 100+ short messages), every turn at just 4 authors
was a full cold read (140–170 s to first token), while about 94% of KV pages were free.

On the same four Sparks and the same simulator, an SGLang TP4 build of the same model held 88% reuse and a 0.56 s
reply start at 16 authors. It overloaded at 24, against 16 for Flare.

Related observation: fully cached long prompts with many messages

A ~208K-token prompt with ~200 messages, reported fully cached (208,030 of 208,047 tokens), still took about 30 s to
first token per question. A ~192K prompt with ~80 messages took 7.2 s. With few messages, warm TTFT is 0.29 s at
10K and 5.8 s at 169K. Cold prefill also slows with message count, at ~14K tokens:

Messages Cold TTFT Warm TTFT
4 6.4 s 0.4 s
30 10.4 s 0.6 s
120 25.7 s 1.0 s
240 44.0 s 1.7 s

That suggests a per-snapshot cost on publish and restore.

Questions
  1. Is 128 a fixed cap, or derived from something we can tune on a multi-host package? It didn't change with the KV
    limit.
  2. Could the slot count scale with the KV pool on large-pool packages? With 25M tokens, memory isn't the constraint
    for this workload.
  3. Is per-snapshot publish/restore cost expected to grow with message count as above?

Multi-host disk prefix caching just landed in #29 (closing #23) and should soften this for returning users; we haven't tested it yet. Raising the GPU slot count would still help. We're happy to rerun the probes against
a branch.

Ngôn ngữ chính
Rust
Star
800
Fork
106
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
100

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/hyperflare-inference

Tất cả issue của NVIDIA/hyperflare-inference

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.