[Engine] RTX 5090 32 GB, Q2_0: the ~13 KB/token KV figure measured, and why 1M context cannot start at --kv int8 (--kv-resident is the lever)
Los mantenedores suelen responder en 4 días
@XiaoBinGan ya está trabajando en esto.
Desde el 12/11/2026.
Evaluación
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Aptitud para principiantes
- 72/100
- Tipo de issue
- Documentación
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- cpp
- Área
- documentation
Línea de trabajo
Lee la sección «¿Cabría?» en docs/MODELS.md y compara sus indicaciones con las mediciones y sugerencias de este issue. Documenta las indicaciones de --kv-resident para tarjetas de 32 GB, la limitación de --kv int8 con un contexto de 1M y las cifras medidas de memoria KV; estará terminado cuando esa sección incluya claramente estos puntos sin insinuar comparaciones de hardware o modelos que no estén respaldadas.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
What this is
Measured on hardware I have here and that I could not find reported for 0.2.13: an RTX 5090 (32 GB, sm_120) with two Xeon Gold 5218 (no AVX-512, PCIe Gen3 x16), 125.6 GiB RAM, Qwen3.8-Flash-Next Q2_0, engine 0.1.47, Docker (image 0.2.2-cuda13), one request at a time.
Two things came out of it that I think are worth a line in the docs, because both are silent:
- The declared
~13 KB per tokenfor the KV cache is what this machine measures — 12.3 to 12.4 KiB/token with--kv int8, half that with--kv q4_0. Numbers below. - At 1M context with
--kv int8the engine starts and then dies, and the line that explains it is buried: the session's own VRAM (mostly its KV paging) leaves the expert cache at 0 slots, then the prompt path cannot allocate. The server exits.--kv-resident 16384is the one flag that fixes the shape of this on a 32 GB card, and it is not mentioned as a 32 GB lever anywhere I could find.
Measured: the KV cache is in RAM, and it costs what the docs say
--kv-resident N keeps the whole KV in pinned host RAM and only the N-token window in VRAM. From the engine's own start line (the K/V in ... of pinned RAM):
| context | --kv |
--kv-resident |
K/V in pinned RAM | KiB / token |
|---|---|---|---|---|
| 65,536 | int8 | 32,768 | 0.77 GiB | 12.3 |
| 262,144 | int8 | 16,384 | 3.09 GiB | 12.4 |
| 524,288 | q4_0 | 16,384 | 3.38 GiB | 6.9 |
| 1,048,576 | q4_0 | 32,768 | 6.75 GiB | 6.9 |
| 1,048,576 | int8 | 32,768 | 12.38 GiB | 12.4 |
So docs/DETAILS.md's "~13 KB for the KV cache and ~0.26 KB for the rope tables per token" is right for int8 on this model, and q4_0 halves it. Nothing surprising — but it does mean a 1M context on this card costs 12.4 GiB of RAM before anything else, which is the cheap part. The VRAM is what breaks.
Measured: what breaks at 1M context on a 32 GB card
--context 1m --kv int8 --vision gpu, the rest as setup writes it:
strata generate: KV streaming: 32768 of 1048576 cells per QSA layer in VRAM, the K/V in 12.38 GiB of pinned RAM
strata generate: rope scaling yarn, factor 4 (freq_scale 0.25, base 1e+07, mscale 1.138629), --max-context 1048576 against a trained context of 262144
strata generate: expert cache auto: 0.52 GiB free, 700 MiB reserved (+184 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 1336 slots (1761 MiB), about 2117 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 2326 MiB at 1048576 tokens), ...
strata serve: prefill: device buffers for a chunk of 1024 tokens do not fit (1 of 32109 MiB free at the failure; 228 MiB taken, the failing buffer wanted 16 MiB): trying a 512-token chunk
strata serve: prefill: device buffers for a chunk of 512 tokens do not fit (1 of 32109 MiB free at the failure; 162 MiB taken, the failing buffer wanted 16 MiB)
strata serve: the GPU has too little free VRAM for the prompt path: turn images off (setup: --vision no), close other programs using the GPU, use a shorter context, or read prompts in smaller chunks (--prefill 512)
strata: The server did not become ready (failed: server_exited).
The engine's own advice names three levers (smaller context, --kv q4_0, smaller --vram-reserve-mib) and the prompt-path message names a fourth (--prefill). What actually moved the needle here was a fifth: --kv-resident. It is the only flag that changes how much of the session sits in VRAM.
Measured: --kv-resident is worth 8x the expert cache on a 32 GB card
Same model, same card, same 1M context, same --kv q4_0, only --kv-resident and what else happened to be on the card changed:
--kv-resident |
VRAM window | expert cache | decode |
|---|---|---|---|
| 32,768 (setup's value at 64K) | 32,768 cells/layer | 1,147 slots | 76.7 tok/s |
| 16,384 | 16,384 cells/layer | 1,209 slots | 82.5 tok/s |
and with the same 16,384 window on a card that had another 5.5 GB free (a co-tenant process stopped), same flags:
| expert cache | VRAM free after load | decode (200-token answers) |
|---|---|---|
| 7,302 slots (10.27 GiB) | 408 MiB | 93.8 tok/s |
That is the whole story on this machine: the KV window is what starves the expert cache, and the expert cache is what the CPU ends up covering for. With 897–1,147 slots the decode hit rate sits at 48–58%, and the engine keeps 30 CPU workers busy (1,011–3,000% CPU on this 2x16-core box, pool workers: 30); with 7,302 slots it reaches 92–93% and the same answers come out faster.
--pool-cores confirms the trade is real but expensive: pinning the pool to 12 logical processors took CPU from 30.7 to 12.0 cores and decode from 76.7 to 34.8 tok/s — halving the CPU halves the speed, because those threads are doing the work the cache cannot.
Measured: prompt speed at long contexts (the card itself, no CPU pool tricks)
| prompt | prompt speed | decode |
|---|---|---|
| 1,176 tok | 596 tok/s | 94.5 tok/s |
| 11,176 tok | 831 tok/s | 85.9 tok/s |
| 44,506 tok | 825 tok/s | 116.0 tok/s |
| 100,628 tok | 754 tok/s | 124.6 tok/s |
100K tokens through a 1M-context session on one 5090, PCIe Gen3, no split: 133.5 s end to end, 6 checkpoints, and the KV hit rate stayed at 80% with 205 MiB read from RAM over the whole request. That is the number I would have wanted to see before trying it.
Suggestion (docs, not code)
- In
docs/MODELS.md→ Will it fit?, one line for the 32 GB cards:--kv-resident 16384is the lever that keeps the expert cache alive at 256K and above; the default 32,768 cells is fine at 64K and wasteful at 1M. - In the same place, say that at
--context 1mwith--kv int8a 32 GB card cannot start at all (session 2,326 MiB + no room for the minimum 1,336 slots), so the honest options are--kv q4_0(6.75 GiB RAM, starts, 1,147 slots) or a smaller context. ~13 KB/tokenverified, and it halves with--kv q4_0— that table above is the whole cost model.
What I could not do
- Only one card and one model: no 24 GB or 48 GB comparison, no IQ3_S numbers, no
--vision gpu+ long context together (the picture reader takes ~1.6 GB of VRAM and was off for the 1M runs). - No
parallel > 1: these are all single-request numbers. - The 1M runs used YaRN factor 4 (past the model's 262,144 training context), so the answers there are not something I am vouching for, only the memory and speed.
- The card was shared with another tenant's job for part of the session, which is why one row above says "card that had another 5.5 GB free" rather than a flag.
- Lenguaje dominante
- C++
- Estrellas
- 11.6k
- Forks
- 1k
- Merge medio
- 18 d 13 h
- PR fusionados (30 d)
- 4
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de Niko1221/Strata
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
Niko1221/Strata#6071 · 2 comentarios ·
Los mantenedores suelen responder en 4 días
-
needs-format
Dificultad 1/5 Menos de una hora Aptitud para principiantes 78/100
Niko1221/Strata#6040 · 1 comentario ·
Los mantenedores suelen responder en 4 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
Los mantenedores suelen responder en 4 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
Los mantenedores suelen responder en 4 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 80/100
Los mantenedores suelen responder en 4 días
Todos los issues de Niko1221/Strata
Issues similares
-
Documentation: Fix parameter markup, legacy branch link, and phrasing in Detachable Joints and Model Command tutorialsPosiblemente ocupada @patelshivani2283-lab la tomó hoy. Abierto
Dificultad 1/5 1-3 horas Aptitud para principiantes 88/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
KhronosGroup/Vulkan-Tutorial#551 ·
Los mantenedores suelen responder en 1 día
-
pumpAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 68/100
dewenni/ESP_Buderus_KM271#191 ·