[Engine] RTX 5090 32 GB, Q2_0: the ~13 KB/token KV figure measured, and why 1M context cannot start at --kv int8 (--kv-resident is the lever)

オープン 初心者向け
#5,852 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 4 日以内に返信

@XiaoBinGan がすでに取り組んでいます。

2026年11月12日 から。

  • #5875 @XiaoBinGan による — オープン
  • #6046 @gputier による — オープン

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
72/100
issue の種類
ドキュメント
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
cpp
領域
documentation

調査の方向性

docs/MODELS.md の「入るか?」セクションを読み、そのガイダンスをこの issue の測定値や提案と比較してください。32 GB カード向けの --kv-resident のガイダンス、1M コンテキストでの --kv int8 の制限、測定された KV メモリ値を記載してください。このセクションにこれらの点が明確に含まれ、根拠のないハードウェアやモデルの比較を示唆していなければ完了です。

索引モデルが issue の本文から書いたものです。

説明

What this is

Measured on hardware I have here and that I could not find reported for 0.2.13: an RTX 5090 (32 GB, sm_120) with two Xeon Gold 5218 (no AVX-512, PCIe Gen3 x16), 125.6 GiB RAM, Qwen3.8-Flash-Next Q2_0, engine 0.1.47, Docker (image 0.2.2-cuda13), one request at a time.

Two things came out of it that I think are worth a line in the docs, because both are silent:

  1. The declared ~13 KB per token for the KV cache is what this machine measures — 12.3 to 12.4 KiB/token with --kv int8, half that with --kv q4_0. Numbers below.
  2. At 1M context with --kv int8 the engine starts and then dies, and the line that explains it is buried: the session's own VRAM (mostly its KV paging) leaves the expert cache at 0 slots, then the prompt path cannot allocate. The server exits. --kv-resident 16384 is the one flag that fixes the shape of this on a 32 GB card, and it is not mentioned as a 32 GB lever anywhere I could find.

Measured: the KV cache is in RAM, and it costs what the docs say

--kv-resident N keeps the whole KV in pinned host RAM and only the N-token window in VRAM. From the engine's own start line (the K/V in ... of pinned RAM):

context --kv --kv-resident K/V in pinned RAM KiB / token
65,536 int8 32,768 0.77 GiB 12.3
262,144 int8 16,384 3.09 GiB 12.4
524,288 q4_0 16,384 3.38 GiB 6.9
1,048,576 q4_0 32,768 6.75 GiB 6.9
1,048,576 int8 32,768 12.38 GiB 12.4

So docs/DETAILS.md's "~13 KB for the KV cache and ~0.26 KB for the rope tables per token" is right for int8 on this model, and q4_0 halves it. Nothing surprising — but it does mean a 1M context on this card costs 12.4 GiB of RAM before anything else, which is the cheap part. The VRAM is what breaks.

Measured: what breaks at 1M context on a 32 GB card

--context 1m --kv int8 --vision gpu, the rest as setup writes it:

strata generate: KV streaming: 32768 of 1048576 cells per QSA layer in VRAM, the K/V in 12.38 GiB of pinned RAM
strata generate: rope scaling yarn, factor 4 (freq_scale 0.25, base 1e+07, mscale 1.138629), --max-context 1048576 against a trained context of 262144
strata generate: expert cache auto: 0.52 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 1336 slots (1761 MiB), about 2117 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 2326 MiB at 1048576 tokens), ...
strata serve: prefill: device buffers for a chunk of 1024 tokens do not fit (1 of 32109 MiB free at the failure; 228 MiB taken, the failing buffer wanted 16 MiB): trying a 512-token chunk
strata serve: prefill: device buffers for a chunk of 512 tokens do not fit (1 of 32109 MiB free at the failure; 162 MiB taken, the failing buffer wanted 16 MiB)
strata serve: the GPU has too little free VRAM for the prompt path: turn images off (setup: --vision no), close other programs using the GPU, use a shorter context, or read prompts in smaller chunks (--prefill 512)
strata: The server did not become ready (failed: server_exited).

The engine's own advice names three levers (smaller context, --kv q4_0, smaller --vram-reserve-mib) and the prompt-path message names a fourth (--prefill). What actually moved the needle here was a fifth: --kv-resident. It is the only flag that changes how much of the session sits in VRAM.

Measured: --kv-resident is worth 8x the expert cache on a 32 GB card

Same model, same card, same 1M context, same --kv q4_0, only --kv-resident and what else happened to be on the card changed:

--kv-resident VRAM window expert cache decode
32,768 (setup's value at 64K) 32,768 cells/layer 1,147 slots 76.7 tok/s
16,384 16,384 cells/layer 1,209 slots 82.5 tok/s

and with the same 16,384 window on a card that had another 5.5 GB free (a co-tenant process stopped), same flags:

expert cache VRAM free after load decode (200-token answers)
7,302 slots (10.27 GiB) 408 MiB 93.8 tok/s

That is the whole story on this machine: the KV window is what starves the expert cache, and the expert cache is what the CPU ends up covering for. With 897–1,147 slots the decode hit rate sits at 48–58%, and the engine keeps 30 CPU workers busy (1,011–3,000% CPU on this 2x16-core box, pool workers: 30); with 7,302 slots it reaches 92–93% and the same answers come out faster.

--pool-cores confirms the trade is real but expensive: pinning the pool to 12 logical processors took CPU from 30.7 to 12.0 cores and decode from 76.7 to 34.8 tok/s — halving the CPU halves the speed, because those threads are doing the work the cache cannot.

Measured: prompt speed at long contexts (the card itself, no CPU pool tricks)

prompt prompt speed decode
1,176 tok 596 tok/s 94.5 tok/s
11,176 tok 831 tok/s 85.9 tok/s
44,506 tok 825 tok/s 116.0 tok/s
100,628 tok 754 tok/s 124.6 tok/s

100K tokens through a 1M-context session on one 5090, PCIe Gen3, no split: 133.5 s end to end, 6 checkpoints, and the KV hit rate stayed at 80% with 205 MiB read from RAM over the whole request. That is the number I would have wanted to see before trying it.

Suggestion (docs, not code)

  • In docs/MODELS.md → Will it fit?, one line for the 32 GB cards: --kv-resident 16384 is the lever that keeps the expert cache alive at 256K and above; the default 32,768 cells is fine at 64K and wasteful at 1M.
  • In the same place, say that at --context 1m with --kv int8 a 32 GB card cannot start at all (session 2,326 MiB + no room for the minimum 1,336 slots), so the honest options are --kv q4_0 (6.75 GiB RAM, starts, 1,147 slots) or a smaller context.
  • ~13 KB/token verified, and it halves with --kv q4_0 — that table above is the whole cost model.

What I could not do

  • Only one card and one model: no 24 GB or 48 GB comparison, no IQ3_S numbers, no --vision gpu + long context together (the picture reader takes ~1.6 GB of VRAM and was off for the 1M runs).
  • No parallel > 1: these are all single-request numbers.
  • The 1M runs used YaRN factor 4 (past the model's 262,144 training context), so the answers there are not something I am vouching for, only the memory and speed.
  • The card was shared with another tenant's job for part of the session, which is why one row above says "card that had another 5.5 GB free" rather than a flag.
主要言語
C++
スター
11.6k
フォーク
1k
平均マージ
18日 13時間
マージ済み PR(30日)
4

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Niko1221/Strata のほかの issue

Niko1221/Strata の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。