[Benchmark] Qwen3.8-Flash-Next IQ3_XXS on RTX 5070 12GB : Strata 0.2.10 vs 0.2.12

The issue contains benchmark results for Strata 0.2.12 and links earlier results in issue #5166; it also names tools/strata_bench.py and the external mtp-bench-3.py script. Read issue #5166 and compare its measurements with the reported results, noting that the engine versions differ and correctness was not tested. Done means the benchmark record is clear and comparable; no code change is specified.

Short version: no regressions, no improvements.
Long version: see benchmarks results for Strata 0.2.12 below.

Benchmarks results for Strata 0.2.10 are in Issue #5166

i7-12700K, 64GB DDR4 (61.3 available), RTX 5070 12GB - pcie gen5, nvme SSD, Linux Ubuntu 26.04, cuda13, Strata 0.2.12
Both CPU and GPU undervolted.
mtp-bench-3.py from PR #4877

tools/strata_bench.py

Community benchmark on NVIDIA GeForce RTX 5070

Measured on 2026-11-07 with tools/strata_bench.py (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

Hardware and software

• GPU and VRAM: NVIDIA GeForce RTX 5070, 12 GiB VRAM
• CPU: 12th Gen Intel(R) Core(TM) i7-12700K (12 cores, 20 threads); installed RAM: 61.3 GiB; storage: nvme0n1 WDS100T1X0E-00AFY0 nvme; nvme2n1 WD Blue SN570 500GB nvme; nvme3n1 WDS100T1X0E-00AFY0 nvme; nvme1n1 WDS200T1X0E-00AFY0 nvme
• OS: Linux 7.0.0-43-generic; GPU driver: 620.70.06; CUDA (driver supports up to): 13.5
• Strata: project ?, commit not a git checkout; engine 0.1.58
• GPU detail (after the run, near idle): power_limit_w=300.00, power_draw_w=136.06, pcie=gen5 x16, pstate=P1, temp_c=54.63
• Background workloads: not recorded (GPU power limit/draw read above)

Model and configuration

• Model and quantization: qwen3.8-flash-next-iq3_xxs
• Context: 192000 tokens; parallel slots: 1
• Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

Method

Run by python tools/strata_bench.py against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

• short chat: the three questions of tools/ab_engine.py, up to 100 new tokens, 3 runs
• 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
• agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (cached_tokens) and timings (prompt_n)
• concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

Results

Ranges are the lowest and highest of the runs; Runs counts requests that returned a token.

Agent session (conversation cache)

Cache hit over the follow-up turns: 70.8 % (70.7-71.0); earlier-turn tokens read again per session: 73 (2-74) (earlier turns that the cache did not hold; the 8207 (8205-8207) new tokens of the files are read once); first turn (nothing cached) first token: 1.21 s (1.13-1.25).

Per-request data: results.json. Engine log lines from the start: engine-log-excerpt.txt (when this run started the server).

Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, nvidia-smi/rocm-smi and the OS; background load is not recorded. Total time of the suite: 63 s.

mtp-bench-3.py

~/AI/Benchmarks/mtp-bench-3.py --url http://127.0.0.1:8090
==============================================================================
MTP benchmark — subject under test
Inference engine : Strata  (version 0.1.58)
LLM / model      : qwen3.8-flash-next-iq3_xxs
Request dialect  : strata-chat
Model path       : /home/andrew/projects/Strata-data/models/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
Endpoint         : http://127.0.0.1:8090
Engine project   : https://github.com/Niko1221/Strata
Engine settings  : drafting=mtp, spec=12, mtp_max=4, lookup=3, spec_min_p=0.50, kv=int8, context=192000, expert_slots=2918, pool_workers=11, gpu_name=NVIDIA GeForce RTX 5070
note: /health: service=strata model=qwen3.8-flash-next-iq3_xxs max_context=192000 loaded=True
note: request dialect: POST /v1/chat/completions (Strata has no llama.cpp POST /completion — that route 404s)
note: drafting method: mtp (spec=12 mtp_max=4 lookup=3)
==============================================================================
Running 9 prompts (max_tokens=192, temperature=0.0, seed=42) via strata-chat

  code_python        pred= 192 draft= 156 acc= 118 rate=0.756 tok/s=133.3
  code_cpp           pred= 192 draft= 135 acc= 105 rate=0.778 tok/s=133.3
  explain_concept    pred= 192 draft= 141 acc= 107 rate=0.759 tok/s=130.6
  summarize          pred= 175 draft= 145 acc= 119 rate=0.821 tok/s=155.1
  qa_factual         pred= 148 draft= 149 acc=  93 rate=0.624 tok/s=140.0
  translation        pred= 192 draft= 148 acc= 106 rate=0.716 tok/s=137.8
  creative_short     pred=  78 draft=  57 acc=  40 rate=0.702 tok/s=123.9
  stepwise_math      pred= 192 draft= 153 acc= 122 rate=0.797 tok/s=147.2
  long_code_review   pred= 192 draft= 175 acc= 119 rate=0.680 tok/s=137.2

Benchmark results — engine: Strata 0.1.58 | LLM: qwen3.8-flash-next-iq3_xxs | dialect: strata-chat
Aggregate: {
  "n_requests": 9,
  "total_predicted": 1553,
  "total_draft": 1259,
  "total_draft_accepted": 929,
  "aggregate_accept_rate": 0.7379,
  "wall_s_total": 16.36,
  "total_cache_n": 0,
  "decode_expert_hit_rate_mean": 0.9462,
  "decode_expert_hit_rate_requests": 9,
  "decode_expert_hit_rate_min": 0.929,
  "decode_expert_hit_rate_max": 0.965,
  "decode_expert_hit_rate_weighted": 0.9458,
  "expert_pcie_share_mean": 0.0064,
  "server_delta": {
    "requests": 9,
    "output_tokens": 1553,
    "drafts_offered": 1259,
    "drafts_accepted": 929,
    "reused": 0,
    "prompt_ms": 5073.3,
    "decode_ms": 11263.9,
    "server_accept_rate": 0.7379,
    "server_decode_tok_s": 137.9
  },
  "per_request_accept_rate": 0.7379,
  "server_accept_rate": 0.7379
}
Aggregate decode expert cache hit rate: 94.6%
(token-weighted 94.6% | per-request 92.9%-96.5% | experts read over PCIe 0.64%)

Expert cache: 2918 expert slots / 4887 MiB VRAM
note: per-request draft counters matches server counters (server: 929->929 accepted of 1259->1259 offered)