Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Community Report: Q4_XS on Multi-GPU NVIDIA

Abierto Apto para principiantes
  • ¿Algo no está bien?
#2,129 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
1/5
Tiempo estimado
Menos de una hora
Aptitud para principiantes
68/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp
Área
backend, performance

Línea de trabajo

Este es un informe detallado de rendimiento de Strata 0.1.43 en el commit 7695859a. Empieza por consultar la documentación de benchmarks del proyecto y tools/strata_bench.py --long, que se usaron en el informe. Compara los resultados indicados del procesamiento de prompts con UD-IQ4_XS e IQ3_XXS en la misma configuración de cuatro GPU y documenta los hallazgos si el proyecto registra informes de hardware. El issue incluye resultados de benchmarks, pero no solicita un cambio de código específico.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Field report: UD-IQ4_XS on NVIDIA (first NVIDIA run, I think) — 4x RTX (5060 Ti + 2x 3090 + 3060), 262K context, mapped mode, 55 GiB RAM budget

Headline: it runs, it is stable, decode is fine (91-96 tok/s), and prompt processing is 2.4-2.9x slower than the same
rig's IQ3_XXS arm (520 / 926 / 1,270 tok/s at 4K / 16K / 32K vs 1,296 / 2,690 / 3,228). For an agent client that
resends a 90-130k-token conversation every turn, that is the whole decision and we are keeping IQ3_XXS. Two full
tools/strata_bench.py --long runs, 3 runs per test, medians below; run 1 and run 2 agree within 0.3% on every prompt
arm. No correctness suite was run.

Hardware and software

  • GPUs: RTX 5060 Ti 16 GB + RTX 3090 24 GB + RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM, all four selected
  • PCIe: the 5060 Ti probes 13.9 GB/s (x8); both 3090s and the 3060 probe 3.1 GB/s (an Oculink/x4 riser and a
    x4-shaped slot). The engine prices an expert miss at x6.35 on three of the four cards.
  • Power: both 3090s capped at 225 W, the 3060 at 125 W (the cap binds during generation).
  • CPU / RAM / disk: Xeon W-1350 @ 3.30 GHz (6C/12T), 123.3 GiB RAM, NVMe.
  • Linux, driver 595.91.07, CUDA 13.2. Strata project 0.1.43, commit 7695859a, engine 0.1.43.
  • Model: unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS, revision 38bb39e, three shards 93.7 GB, all three SHA-256
    match the table in docs/UNSLOTH_Q4.md.

Configuration

strata --serve --pack packs/ud-iq4_xs
       --native .../Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
       --resident-budget-gib 55 --expert-profile data/expert-profile.bin --expert-cache auto
       --prefill auto --spec 4 --mtp mtp/rt --max-context 262144 --kv q4_0 --vision
       --vram-reserve-mib 700 --spec-min-p 0.5 --layer-split auto

STRATA_PF_FUSED=1, one parallel slot, thinking off, temperature 0, loading not timed. No --ple-gguf: shard 1 holds
0 tensors and shard 2 holds per_layer_token_embd.weight (IQ4_NL, [160, 320001536]), which the engine found by name,
as documented. Vision encoder on the 5060 Ti; no image was sent (the doc says images are untested on this pack).

Pack built with tools/iq_pack.py --compat-bf16: 1,079 tensors, 303 served natively, arena 1.38 GiB, 460
conversions (264 exact, 196 rounded, max |err| 0.0144), native_experts.txt v4 with the per-role
gate,up,down shard column on 14 rows, expert arena 59,519,795,200 bytes (55.4 GiB). tools/test_iq_pack.py
24/24 OK. The --compat-bf16 rounding is on, so the 6-9% PPL penalty the doc notes on Strix Halo applies to this arm too.

What the engine chose (this is where the story is)

this pack our IQ3_XXS arm, same four cards
split chosen by auto K=3,11,30 (5060 Ti takes layers 3-10, 3,715 slots) K=3,6,30 (5060 Ti takes 3 layers)
measured decode — chat / agent tok/s 90.9 / 96.1 (11.0 / 10.4 ms per token) 111.3 / 109.8 (9.0 / 9.1 ms)
expert pairs cached 20,132 of 24,576 (~99.2% of the routed mass) 24,387 of 24,576
experts resident 81% 98%
RAM tier 24.23 GiB page-locked + mapped in 16 s; 18,388 experts left on the mmap fallback none (plain mode)
prompt chunk 7,424 tokens, 96-slot ring, 5,566 cache slots borrowed 4,096 tokens
decode cache hit 98.0-99.8%, 0 blob reads from the file 99.9-100.0%

So decode was in good shape: 81% resident, ~99% hit rate, nothing read from the SSD while generating. The prompt path
is the one that suffers, and it is not a chunk clamp (this arm got a bigger chunk than IQ3_XXS). Reading a prompt
streams every expert once per chunk, and 19% of them are not in VRAM on three cards whose links probe 3.1 GB/s.

Results — run 2 (medians, range over 3 runs; 9 requests for the chat arm)

test prompt tokens reused generated prompt tok/s decode tok/s TTFT (s)
short chat 46 (43-51) 0 100 — 90.9 (84.4-121.3) 0.30 (0.28-0.34)
4K prompt, fresh 4,019 0 24 cap 519.8 (519.7-519.8) — 7.75
16K prompt, fresh 16,058 0 24 cap 925.5 (925.3-925.7) — 17.38
32K prompt, fresh 32,049 0 24 cap 1,269.5 (1,268.9-1,269.9) — 25.28
agent session, follow-up turns 14,494 at the last turn see below ≤ 64 see below 96.1 (77.0-115.5) 6.88 (5.52-7.32)

Agent cache hit 68.8% (68.8-68.9), earlier-turn tokens re-read per session 2 (2-4), first turn (nothing cached)
TTFT 3.27 s. Suite total 292 s.

Run 1 vs run 2

metric run 1 run 2
4K prompt tok/s 518.1 519.8
16K prompt tok/s 923.9 925.5
32K prompt tok/s 1,268.3 1,269.5
chat decode tok/s 87.3 (61.8-129.6) 90.9 (84.4-121.3)
chat first token (s) 0.35 0.30
agent decode tok/s 88.1 96.1
agent turn TTFT (s) 6.88 6.88

Prefill is reproducible to 0.3%. Decode wanders ±8% run to run, same as our other arms on this box.

Head to head with the same rig's IQ3_XXS arm (same suite, same day, both at layer_split auto's own optimum)

IQ3_XXS (plain) UD-IQ4_XS (mapped, 55 GiB) delta
4K prompt tok/s 1,295.9 519.8 -60%
16K prompt tok/s 2,689.9 925.5 -66%
32K prompt tok/s 3,227.8 1,269.5 -61%
chat decode tok/s 111.3 90.9 -18%
agent decode tok/s 109.8 96.1 -12%
TTFT at 32K 9.96 s 25.28 s +154%
agent turn TTFT 2.61 s 6.88 s +164%
chat TTFT 0.30 s 0.30 s 0

A 100k-token agent turn with a 1k-token answer goes from ~40 s to ~80 s. The +39% of expert bytes
(42.9 → 55.4 GiB) does not fit in 76 GiB of VRAM next to a 262K q4_0 KV cache, and on this box a miss costs 6.4x
what the engine's default pcie_frac assumes.

Not measured

Correctness (speed suite only; needle probes not run on this pack), prompts past 32K, --clients concurrency (one
slot), images, sampled decoding, and any comparison against llama.cpp on the same file. --compat-bf16 rounding is on,
so quality is the pack's, not the GGUF's. Storage model and CPU temperature not recorded.

Lenguaje dominante
C++
Estrellas
11.6k
Forks
1k
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Niko1221/Strata

Todos los issues de Niko1221/Strata

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.