[Engine] RTX 5090 32 GB, Q2_0: the ~13 KB/token KV figure measured, and why 1M context cannot start at --kv int8 (--kv-resident is the lever)

Aperta Adatta ai principianti
#5,852 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 4 giorni

@XiaoBinGan ci sta già lavorando.

Dal 12/11/2026.

  • #5875 di @XiaoBinGan — aperta
  • #6046 di @gputier — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
72/100
Tipo di issue
Documentazione
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
documentation

Direzione di ricerca

Leggi la sezione «Ci starà?» in docs/MODELS.md e confronta le indicazioni con le misurazioni e i suggerimenti di questa issue. Documenta le indicazioni di --kv-resident per le schede da 32 GB, il limite di --kv int8 con un contesto di 1M e i valori misurati della memoria KV; il lavoro è concluso quando questa sezione include chiaramente questi punti senza suggerire confronti tra hardware o modelli non supportati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What this is

Measured on hardware I have here and that I could not find reported for 0.2.13: an RTX 5090 (32 GB, sm_120) with two Xeon Gold 5218 (no AVX-512, PCIe Gen3 x16), 125.6 GiB RAM, Qwen3.8-Flash-Next Q2_0, engine 0.1.47, Docker (image 0.2.2-cuda13), one request at a time.

Two things came out of it that I think are worth a line in the docs, because both are silent:

  1. The declared ~13 KB per token for the KV cache is what this machine measures — 12.3 to 12.4 KiB/token with --kv int8, half that with --kv q4_0. Numbers below.
  2. At 1M context with --kv int8 the engine starts and then dies, and the line that explains it is buried: the session's own VRAM (mostly its KV paging) leaves the expert cache at 0 slots, then the prompt path cannot allocate. The server exits. --kv-resident 16384 is the one flag that fixes the shape of this on a 32 GB card, and it is not mentioned as a 32 GB lever anywhere I could find.

Measured: the KV cache is in RAM, and it costs what the docs say

--kv-resident N keeps the whole KV in pinned host RAM and only the N-token window in VRAM. From the engine's own start line (the K/V in ... of pinned RAM):

context --kv --kv-resident K/V in pinned RAM KiB / token
65,536 int8 32,768 0.77 GiB 12.3
262,144 int8 16,384 3.09 GiB 12.4
524,288 q4_0 16,384 3.38 GiB 6.9
1,048,576 q4_0 32,768 6.75 GiB 6.9
1,048,576 int8 32,768 12.38 GiB 12.4

So docs/DETAILS.md's "~13 KB for the KV cache and ~0.26 KB for the rope tables per token" is right for int8 on this model, and q4_0 halves it. Nothing surprising — but it does mean a 1M context on this card costs 12.4 GiB of RAM before anything else, which is the cheap part. The VRAM is what breaks.

Measured: what breaks at 1M context on a 32 GB card

--context 1m --kv int8 --vision gpu, the rest as setup writes it:

strata generate: KV streaming: 32768 of 1048576 cells per QSA layer in VRAM, the K/V in 12.38 GiB of pinned RAM
strata generate: rope scaling yarn, factor 4 (freq_scale 0.25, base 1e+07, mscale 1.138629), --max-context 1048576 against a trained context of 262144
strata generate: expert cache auto: 0.52 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 1336 slots (1761 MiB), about 2117 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 2326 MiB at 1048576 tokens), ...
strata serve: prefill: device buffers for a chunk of 1024 tokens do not fit (1 of 32109 MiB free at the failure; 228 MiB taken, the failing buffer wanted 16 MiB): trying a 512-token chunk
strata serve: prefill: device buffers for a chunk of 512 tokens do not fit (1 of 32109 MiB free at the failure; 162 MiB taken, the failing buffer wanted 16 MiB)
strata serve: the GPU has too little free VRAM for the prompt path: turn images off (setup: --vision no), close other programs using the GPU, use a shorter context, or read prompts in smaller chunks (--prefill 512)
strata: The server did not become ready (failed: server_exited).

The engine's own advice names three levers (smaller context, --kv q4_0, smaller --vram-reserve-mib) and the prompt-path message names a fourth (--prefill). What actually moved the needle here was a fifth: --kv-resident. It is the only flag that changes how much of the session sits in VRAM.

Measured: --kv-resident is worth 8x the expert cache on a 32 GB card

Same model, same card, same 1M context, same --kv q4_0, only --kv-resident and what else happened to be on the card changed:

--kv-resident VRAM window expert cache decode
32,768 (setup's value at 64K) 32,768 cells/layer 1,147 slots 76.7 tok/s
16,384 16,384 cells/layer 1,209 slots 82.5 tok/s

and with the same 16,384 window on a card that had another 5.5 GB free (a co-tenant process stopped), same flags:

expert cache VRAM free after load decode (200-token answers)
7,302 slots (10.27 GiB) 408 MiB 93.8 tok/s

That is the whole story on this machine: the KV window is what starves the expert cache, and the expert cache is what the CPU ends up covering for. With 897–1,147 slots the decode hit rate sits at 48–58%, and the engine keeps 30 CPU workers busy (1,011–3,000% CPU on this 2x16-core box, pool workers: 30); with 7,302 slots it reaches 92–93% and the same answers come out faster.

--pool-cores confirms the trade is real but expensive: pinning the pool to 12 logical processors took CPU from 30.7 to 12.0 cores and decode from 76.7 to 34.8 tok/s — halving the CPU halves the speed, because those threads are doing the work the cache cannot.

Measured: prompt speed at long contexts (the card itself, no CPU pool tricks)

prompt prompt speed decode
1,176 tok 596 tok/s 94.5 tok/s
11,176 tok 831 tok/s 85.9 tok/s
44,506 tok 825 tok/s 116.0 tok/s
100,628 tok 754 tok/s 124.6 tok/s

100K tokens through a 1M-context session on one 5090, PCIe Gen3, no split: 133.5 s end to end, 6 checkpoints, and the KV hit rate stayed at 80% with 205 MiB read from RAM over the whole request. That is the number I would have wanted to see before trying it.

Suggestion (docs, not code)

  • In docs/MODELS.md → Will it fit?, one line for the 32 GB cards: --kv-resident 16384 is the lever that keeps the expert cache alive at 256K and above; the default 32,768 cells is fine at 64K and wasteful at 1M.
  • In the same place, say that at --context 1m with --kv int8 a 32 GB card cannot start at all (session 2,326 MiB + no room for the minimum 1,336 slots), so the honest options are --kv q4_0 (6.75 GiB RAM, starts, 1,147 slots) or a smaller context.
  • ~13 KB/token verified, and it halves with --kv q4_0 — that table above is the whole cost model.

What I could not do

  • Only one card and one model: no 24 GB or 48 GB comparison, no IQ3_S numbers, no --vision gpu + long context together (the picture reader takes ~1.6 GB of VRAM and was off for the 1M runs).
  • No parallel > 1: these are all single-request numbers.
  • The 1M runs used YaRN factor 4 (past the model's 262,144 training context), so the answers there are not something I am vouching for, only the memory and speed.
  • The card was shared with another tenant's job for part of the session, which is why one row above says "card that had another 5.5 GB free" rather than a flag.
Lingua principale
C++
Stelle
11.6k
Fork
1k
Merge medio
18g 13h
PR unite (30g)
4

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Niko1221/Strata

Tutte le issue di Niko1221/Strata

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.