Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Community Report: Q4_XS on Multi-GPU NVIDIA

Open Beginner friendly
  • Is something wrong?
#2,129 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
68/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
backend, performance

Research direction

This is a detailed performance report for Strata 0.1.43 at commit 7695859a; begin by checking the project's benchmark documentation and tools/strata_bench.py --long, which the report used. Compare the reported UD-IQ4_XS and IQ3_XXS prompt processing results on the same four-GPU setup, and document the findings if the project tracks hardware reports. The issue includes benchmark results but does not request a specific code change.

Written by the indexing model from the issue text.

Description

Field report: UD-IQ4_XS on NVIDIA (first NVIDIA run, I think) — 4x RTX (5060 Ti + 2x 3090 + 3060), 262K context, mapped mode, 55 GiB RAM budget

Headline: it runs, it is stable, decode is fine (91-96 tok/s), and prompt processing is 2.4-2.9x slower than the same
rig's IQ3_XXS arm (520 / 926 / 1,270 tok/s at 4K / 16K / 32K vs 1,296 / 2,690 / 3,228). For an agent client that
resends a 90-130k-token conversation every turn, that is the whole decision and we are keeping IQ3_XXS. Two full
tools/strata_bench.py --long runs, 3 runs per test, medians below; run 1 and run 2 agree within 0.3% on every prompt
arm. No correctness suite was run.

Hardware and software

  • GPUs: RTX 5060 Ti 16 GB + RTX 3090 24 GB + RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM, all four selected
  • PCIe: the 5060 Ti probes 13.9 GB/s (x8); both 3090s and the 3060 probe 3.1 GB/s (an Oculink/x4 riser and a
    x4-shaped slot). The engine prices an expert miss at x6.35 on three of the four cards.
  • Power: both 3090s capped at 225 W, the 3060 at 125 W (the cap binds during generation).
  • CPU / RAM / disk: Xeon W-1350 @ 3.30 GHz (6C/12T), 123.3 GiB RAM, NVMe.
  • Linux, driver 595.91.07, CUDA 13.2. Strata project 0.1.43, commit 7695859a, engine 0.1.43.
  • Model: unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS, revision 38bb39e, three shards 93.7 GB, all three SHA-256
    match the table in docs/UNSLOTH_Q4.md.

Configuration

strata --serve --pack packs/ud-iq4_xs
       --native .../Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
       --resident-budget-gib 55 --expert-profile data/expert-profile.bin --expert-cache auto
       --prefill auto --spec 4 --mtp mtp/rt --max-context 262144 --kv q4_0 --vision
       --vram-reserve-mib 700 --spec-min-p 0.5 --layer-split auto

STRATA_PF_FUSED=1, one parallel slot, thinking off, temperature 0, loading not timed. No --ple-gguf: shard 1 holds
0 tensors and shard 2 holds per_layer_token_embd.weight (IQ4_NL, [160, 320001536]), which the engine found by name,
as documented. Vision encoder on the 5060 Ti; no image was sent (the doc says images are untested on this pack).

Pack built with tools/iq_pack.py --compat-bf16: 1,079 tensors, 303 served natively, arena 1.38 GiB, 460
conversions (264 exact, 196 rounded, max |err| 0.0144), native_experts.txt v4 with the per-role
gate,up,down shard column on 14 rows, expert arena 59,519,795,200 bytes (55.4 GiB). tools/test_iq_pack.py
24/24 OK. The --compat-bf16 rounding is on, so the 6-9% PPL penalty the doc notes on Strix Halo applies to this arm too.

What the engine chose (this is where the story is)

this pack our IQ3_XXS arm, same four cards
split chosen by auto K=3,11,30 (5060 Ti takes layers 3-10, 3,715 slots) K=3,6,30 (5060 Ti takes 3 layers)
measured decode — chat / agent tok/s 90.9 / 96.1 (11.0 / 10.4 ms per token) 111.3 / 109.8 (9.0 / 9.1 ms)
expert pairs cached 20,132 of 24,576 (~99.2% of the routed mass) 24,387 of 24,576
experts resident 81% 98%
RAM tier 24.23 GiB page-locked + mapped in 16 s; 18,388 experts left on the mmap fallback none (plain mode)
prompt chunk 7,424 tokens, 96-slot ring, 5,566 cache slots borrowed 4,096 tokens
decode cache hit 98.0-99.8%, 0 blob reads from the file 99.9-100.0%

So decode was in good shape: 81% resident, ~99% hit rate, nothing read from the SSD while generating. The prompt path
is the one that suffers, and it is not a chunk clamp (this arm got a bigger chunk than IQ3_XXS). Reading a prompt
streams every expert once per chunk, and 19% of them are not in VRAM on three cards whose links probe 3.1 GB/s.

Results — run 2 (medians, range over 3 runs; 9 requests for the chat arm)

test prompt tokens reused generated prompt tok/s decode tok/s TTFT (s)
short chat 46 (43-51) 0 100 — 90.9 (84.4-121.3) 0.30 (0.28-0.34)
4K prompt, fresh 4,019 0 24 cap 519.8 (519.7-519.8) — 7.75
16K prompt, fresh 16,058 0 24 cap 925.5 (925.3-925.7) — 17.38
32K prompt, fresh 32,049 0 24 cap 1,269.5 (1,268.9-1,269.9) — 25.28
agent session, follow-up turns 14,494 at the last turn see below ≤ 64 see below 96.1 (77.0-115.5) 6.88 (5.52-7.32)

Agent cache hit 68.8% (68.8-68.9), earlier-turn tokens re-read per session 2 (2-4), first turn (nothing cached)
TTFT 3.27 s. Suite total 292 s.

Run 1 vs run 2

metric run 1 run 2
4K prompt tok/s 518.1 519.8
16K prompt tok/s 923.9 925.5
32K prompt tok/s 1,268.3 1,269.5
chat decode tok/s 87.3 (61.8-129.6) 90.9 (84.4-121.3)
chat first token (s) 0.35 0.30
agent decode tok/s 88.1 96.1
agent turn TTFT (s) 6.88 6.88

Prefill is reproducible to 0.3%. Decode wanders ±8% run to run, same as our other arms on this box.

Head to head with the same rig's IQ3_XXS arm (same suite, same day, both at layer_split auto's own optimum)

IQ3_XXS (plain) UD-IQ4_XS (mapped, 55 GiB) delta
4K prompt tok/s 1,295.9 519.8 -60%
16K prompt tok/s 2,689.9 925.5 -66%
32K prompt tok/s 3,227.8 1,269.5 -61%
chat decode tok/s 111.3 90.9 -18%
agent decode tok/s 109.8 96.1 -12%
TTFT at 32K 9.96 s 25.28 s +154%
agent turn TTFT 2.61 s 6.88 s +164%
chat TTFT 0.30 s 0.30 s 0

A 100k-token agent turn with a 1k-token answer goes from ~40 s to ~80 s. The +39% of expert bytes
(42.9 → 55.4 GiB) does not fit in 76 GiB of VRAM next to a 262K q4_0 KV cache, and on this box a miss costs 6.4x
what the engine's default pcie_frac assumes.

Not measured

Correctness (speed suite only; needle probes not run on this pack), prompts past 32K, --clients concurrency (one
slot), images, sampled decoding, and any comparison against llama.cpp on the same file. --compat-bf16 rounding is on,
so quality is the pack's, not the GGUF's. Storage model and CPU temperature not recorded.

Dominant language
C++
Stars
11.6k
Forks
1k
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Niko1221/Strata

All issues in Niko1221/Strata

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.