[Benchmark] Qwen3.8-Flash-Next-Unsloth-UD-IQ2_XS DFlash2 on RTX 5070 12GB : 105~121 tok/s (Strata 0.2.11)

Questa issue registra i risultati dei benchmark di tools/strata_bench.py e mtp-bench-3.py; inizia leggendo questi script e i dati dei benchmark allegati. Il lavoro è concluso quando i risultati e la configurazione della macchina sono registrati con chiarezza sufficiente per confrontare le prestazioni; l’issue non richiede una modifica al codice.

i7-12700K, 64GB DDR4 (61.3 available), RTX 5070 12GB - pcie gen5, nvme SSD, Linux Ubuntu 26.04, cuda13, Strata 0.2.11
Both CPU and GPU undervolted.
mtp-bench-3.py from PR #4877

tools/strata_bench.py

Community benchmark on NVIDIA GeForce RTX 5070

Measured on 2026-11-06 with tools/strata_bench.py (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

Hardware and software

• GPU and VRAM: NVIDIA GeForce RTX 5070, 12 GiB VRAM
• CPU: 12th Gen Intel(R) Core(TM) i7-12700K (12 cores, 20 threads); installed RAM: 61.3 GiB; storage: nvme2n1 WDS100T1X0E-00AFY0 nvme; nvme1n1 WD Blue SN570 500GB nvme; nvme3n1 WDS100T1X0E-00AFY0 nvme; nvme0n1 WDS200T1X0E-00AFY0 nvme
• OS: Linux 7.0.0-38-generic; GPU driver: 615.71.10; CUDA (driver supports up to): 13.4
• Strata: project ?, commit not a git checkout; engine 0.1.57
• GPU detail (after the run, near idle): power_limit_w=300.00, power_draw_w=138.32, pcie=gen5 x16, pstate=P1, temp_c=58
• Background workloads: not recorded (GPU power limit/draw read above)

Model and configuration

• Model and quantization: qwen3.8-flash-next-unsloth-ud-iq2_xs
• Context: 204800 tokens; parallel slots: 1
• Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

Method

Run by python tools/strata_bench.py against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

• short chat: the three questions of tools/ab_engine.py, up to 100 new tokens, 3 runs
• 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
• agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (cached_tokens) and timings (prompt_n)
• concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

Results

Ranges are the lowest and highest of the runs; Runs counts requests that returned a token.

Agent session (conversation cache)

Cache hit over the follow-up turns: 70.7 % (70.7-70.9); earlier-turn tokens read again per session: 5 (4-74) (earlier turns that the cache did not hold; the 8206 (8206-8207) new tokens of the files are read once); first turn (nothing cached) first token: 1.20 s (1.10-1.24).

Per-request data: results.json. Engine log lines from the start: engine-log-excerpt.txt (when this run started the server).

Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, nvidia-smi/rocm-smi and the OS; background load is not recorded. Total time of the suite: 63 s.

mtp-bench-3.py

~/AI/Benchmarks/mtp-bench-3.py --url http://127.0.0.1:8090
==============================================================================
MTP benchmark — subject under test
Inference engine : Strata  (version 0.1.57)
LLM / model      : qwen3.8-flash-next-unsloth-ud-iq2_xs
Request dialect  : strata-chat
Model path       : /home/andrew/projects/Strata-data/models/models/unsloth-UD-IQ2_XS/Qwen3.8-Flash-Next-UD-IQ2_XS-00001-of-00003.gguf
Endpoint         : http://127.0.0.1:8090
Engine project   : https://github.com/Niko1221/Strata
Engine settings  : drafting=dflash2, spec=6, mtp_max=6, lookup=3, spec_min_p=0.00, kv=int8, context=204800, expert_slots=2978, pool_workers=11, gpu_name=NVIDIA GeForce RTX 5070
note: /health: service=strata model=qwen3.8-flash-next-unsloth-ud-iq2_xs max_context=204800 loaded=True
note: request dialect: POST /v1/chat/completions (Strata has no llama.cpp POST /completion — that route 404s)
note: drafting method: dflash2 (spec=6 mtp_max=6 lookup=3)
==============================================================================
Running 9 prompts (max_tokens=192, temperature=0.0, seed=42) via strata-chat

  code_python        pred= 192 draft= 140 acc= 118 rate=0.843 tok/s=125.9
  code_cpp           pred= 192 draft= 121 acc= 101 rate=0.835 tok/s=116.3
  explain_concept    pred= 192 draft= 105 acc=  94 rate=0.895 tok/s=111.1
  summarize          pred= 174 draft= 148 acc= 118 rate=0.797 tok/s=132.1
  qa_factual         pred=  85 draft=  70 acc=  52 rate=0.743 tok/s=128.3
  translation        pred= 107 draft=  74 acc=  58 rate=0.784 tok/s=122.1
  creative_short     pred=  79 draft=  53 acc=  40 rate=0.755 tok/s=111.8
  stepwise_math      pred= 192 draft= 154 acc= 134 rate=0.870 tok/s=144.1
  long_code_review   pred= 192 draft= 111 acc=  89 rate=0.802 tok/s=106.2

Benchmark results — engine: Strata 0.1.57 | LLM: qwen3.8-flash-next-unsloth-ud-iq2_xs | dialect: strata-chat
Aggregate: {
  "n_requests": 9,
  "total_predicted": 1405,
  "total_draft": 976,
  "total_draft_accepted": 804,
  "aggregate_accept_rate": 0.8238,
  "wall_s_total": 16.74,
  "total_cache_n": 0,
  "decode_expert_hit_rate_mean": 0.9216,
  "decode_expert_hit_rate_requests": 9,
  "decode_expert_hit_rate_min": 0.907,
  "decode_expert_hit_rate_max": 0.938,
  "decode_expert_hit_rate_weighted": 0.9199,
  "expert_pcie_share_mean": 0.0111,
  "server_delta": {
    "requests": 9,
    "output_tokens": 1405,
    "drafts_offered": 976,
    "drafts_accepted": 804,
    "reused": 0,
    "prompt_ms": 5105.6,
    "decode_ms": 11606.9,
    "server_accept_rate": 0.8238,
    "server_decode_tok_s": 121.0
  },
  "per_request_accept_rate": 0.8238,
  "server_accept_rate": 0.8238
}
Aggregate decode expert cache hit rate: 92.2%
(token-weighted 92.0% | per-request 90.7%-93.8% | experts read over PCIe 1.11%)

Expert cache: 2978 expert slots / 4876 MiB VRAM
note: per-request draft counters matches server counters (server: 804->804 accepted of 976->976 offered)