[Engine] RTX 5090 32 GB, Q2_0: the ~13 KB/token KV figure measured, and why 1M context cannot start at --kv int8 (--kv-resident is the lever)
I maintainer di solito rispondono entro 4 giorni
@XiaoBinGan ci sta già lavorando.
Dal 12/11/2026.
Valutazione
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Idoneità per principianti
- 72/100
- Tipo di issue
- Documentazione
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- cpp
- Ambito
- documentation
Direzione di ricerca
Leggi la sezione «Ci starà?» in docs/MODELS.md e confronta le indicazioni con le misurazioni e i suggerimenti di questa issue. Documenta le indicazioni di --kv-resident per le schede da 32 GB, il limite di --kv int8 con un contesto di 1M e i valori misurati della memoria KV; il lavoro è concluso quando questa sezione include chiaramente questi punti senza suggerire confronti tra hardware o modelli non supportati.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
What this is
Measured on hardware I have here and that I could not find reported for 0.2.13: an RTX 5090 (32 GB, sm_120) with two Xeon Gold 5218 (no AVX-512, PCIe Gen3 x16), 125.6 GiB RAM, Qwen3.8-Flash-Next Q2_0, engine 0.1.47, Docker (image 0.2.2-cuda13), one request at a time.
Two things came out of it that I think are worth a line in the docs, because both are silent:
- The declared
~13 KB per tokenfor the KV cache is what this machine measures — 12.3 to 12.4 KiB/token with--kv int8, half that with--kv q4_0. Numbers below. - At 1M context with
--kv int8the engine starts and then dies, and the line that explains it is buried: the session's own VRAM (mostly its KV paging) leaves the expert cache at 0 slots, then the prompt path cannot allocate. The server exits.--kv-resident 16384is the one flag that fixes the shape of this on a 32 GB card, and it is not mentioned as a 32 GB lever anywhere I could find.
Measured: the KV cache is in RAM, and it costs what the docs say
--kv-resident N keeps the whole KV in pinned host RAM and only the N-token window in VRAM. From the engine's own start line (the K/V in ... of pinned RAM):
| context | --kv |
--kv-resident |
K/V in pinned RAM | KiB / token |
|---|---|---|---|---|
| 65,536 | int8 | 32,768 | 0.77 GiB | 12.3 |
| 262,144 | int8 | 16,384 | 3.09 GiB | 12.4 |
| 524,288 | q4_0 | 16,384 | 3.38 GiB | 6.9 |
| 1,048,576 | q4_0 | 32,768 | 6.75 GiB | 6.9 |
| 1,048,576 | int8 | 32,768 | 12.38 GiB | 12.4 |
So docs/DETAILS.md's "~13 KB for the KV cache and ~0.26 KB for the rope tables per token" is right for int8 on this model, and q4_0 halves it. Nothing surprising — but it does mean a 1M context on this card costs 12.4 GiB of RAM before anything else, which is the cheap part. The VRAM is what breaks.
Measured: what breaks at 1M context on a 32 GB card
--context 1m --kv int8 --vision gpu, the rest as setup writes it:
strata generate: KV streaming: 32768 of 1048576 cells per QSA layer in VRAM, the K/V in 12.38 GiB of pinned RAM
strata generate: rope scaling yarn, factor 4 (freq_scale 0.25, base 1e+07, mscale 1.138629), --max-context 1048576 against a trained context of 262144
strata generate: expert cache auto: 0.52 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 1336 slots (1761 MiB), about 2117 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 2326 MiB at 1048576 tokens), ...
strata serve: prefill: device buffers for a chunk of 1024 tokens do not fit (1 of 32109 MiB free at the failure; 228 MiB taken, the failing buffer wanted 16 MiB): trying a 512-token chunk
strata serve: prefill: device buffers for a chunk of 512 tokens do not fit (1 of 32109 MiB free at the failure; 162 MiB taken, the failing buffer wanted 16 MiB)
strata serve: the GPU has too little free VRAM for the prompt path: turn images off (setup: --vision no), close other programs using the GPU, use a shorter context, or read prompts in smaller chunks (--prefill 512)
strata: The server did not become ready (failed: server_exited).
The engine's own advice names three levers (smaller context, --kv q4_0, smaller --vram-reserve-mib) and the prompt-path message names a fourth (--prefill). What actually moved the needle here was a fifth: --kv-resident. It is the only flag that changes how much of the session sits in VRAM.
Measured: --kv-resident is worth 8x the expert cache on a 32 GB card
Same model, same card, same 1M context, same --kv q4_0, only --kv-resident and what else happened to be on the card changed:
--kv-resident |
VRAM window | expert cache | decode |
|---|---|---|---|
| 32,768 (setup's value at 64K) | 32,768 cells/layer | 1,147 slots | 76.7 tok/s |
| 16,384 | 16,384 cells/layer | 1,209 slots | 82.5 tok/s |
and with the same 16,384 window on a card that had another 5.5 GB free (a co-tenant process stopped), same flags:
| expert cache | VRAM free after load | decode (200-token answers) |
|---|---|---|
| 7,302 slots (10.27 GiB) | 408 MiB | 93.8 tok/s |
That is the whole story on this machine: the KV window is what starves the expert cache, and the expert cache is what the CPU ends up covering for. With 897–1,147 slots the decode hit rate sits at 48–58%, and the engine keeps 30 CPU workers busy (1,011–3,000% CPU on this 2x16-core box, pool workers: 30); with 7,302 slots it reaches 92–93% and the same answers come out faster.
--pool-cores confirms the trade is real but expensive: pinning the pool to 12 logical processors took CPU from 30.7 to 12.0 cores and decode from 76.7 to 34.8 tok/s — halving the CPU halves the speed, because those threads are doing the work the cache cannot.
Measured: prompt speed at long contexts (the card itself, no CPU pool tricks)
| prompt | prompt speed | decode |
|---|---|---|
| 1,176 tok | 596 tok/s | 94.5 tok/s |
| 11,176 tok | 831 tok/s | 85.9 tok/s |
| 44,506 tok | 825 tok/s | 116.0 tok/s |
| 100,628 tok | 754 tok/s | 124.6 tok/s |
100K tokens through a 1M-context session on one 5090, PCIe Gen3, no split: 133.5 s end to end, 6 checkpoints, and the KV hit rate stayed at 80% with 205 MiB read from RAM over the whole request. That is the number I would have wanted to see before trying it.
Suggestion (docs, not code)
- In
docs/MODELS.md→ Will it fit?, one line for the 32 GB cards:--kv-resident 16384is the lever that keeps the expert cache alive at 256K and above; the default 32,768 cells is fine at 64K and wasteful at 1M. - In the same place, say that at
--context 1mwith--kv int8a 32 GB card cannot start at all (session 2,326 MiB + no room for the minimum 1,336 slots), so the honest options are--kv q4_0(6.75 GiB RAM, starts, 1,147 slots) or a smaller context. ~13 KB/tokenverified, and it halves with--kv q4_0— that table above is the whole cost model.
What I could not do
- Only one card and one model: no 24 GB or 48 GB comparison, no IQ3_S numbers, no
--vision gpu+ long context together (the picture reader takes ~1.6 GB of VRAM and was off for the 1M runs). - No
parallel > 1: these are all single-request numbers. - The 1M runs used YaRN factor 4 (past the model's 262,144 training context), so the answers there are not something I am vouching for, only the memory and speed.
- The card was shared with another tenant's job for part of the session, which is why one row above says "card that had another 5.5 GB free" rather than a flag.
- Lingua principale
- C++
- Stelle
- 11.6k
- Fork
- 1k
- Merge medio
- 18g 13h
- PR unite (30g)
- 4
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di Niko1221/Strata
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
Niko1221/Strata#6071 · 2 commenti ·
I maintainer di solito rispondono entro 4 giorni
-
needs-format
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 78/100
Niko1221/Strata#6040 · 1 commento ·
I maintainer di solito rispondono entro 4 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
I maintainer di solito rispondono entro 4 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
I maintainer di solito rispondono entro 4 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
I maintainer di solito rispondono entro 4 giorni
Tutte le issue di Niko1221/Strata
Issue simili
-
Documentation: Fix parameter markup, legacy branch link, and phrasing in Detachable Joints and Model Command tutorialsForse già presa @patelshivani2283-lab l’ha presa oggi. Aperta
Difficoltà 1/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
KhronosGroup/Vulkan-Tutorial#551 ·
I maintainer di solito rispondono entro 1 giorno
-
pumpAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 68/100
dewenni/ESP_Buderus_KM271#191 ·