[Engine] Arc Pro B70: the int8 prompt-expert default (STRATA_PREFILL_MMQ unset = auto) reads a 6.3k prompt 15 % slower than the FP16 chain, and its per-start crossover (32 to 154 rows) changes the greedy answer between starts

Start with sycl/src/prefill/prefill.cpp, sycl/include/strata/prefill/esimd_moe.hpp, the strata-sycl.sh launcher, and docs/INTEL_INT8_EXPERTS.md. Reproduce on an Intel Arc Pro B70 with STRATA_PREFILL_MMQ unset, then inspect how crossover timings are measured and cached across container starts. Done means the auto setting gives stable crossover behavior and repeatable prompt results, with the cache persisting between starts.

Area

[Engine]

Desktop app, Terminal or source?

Source

What happened

Updating a B70 install from the v0.1.45 source build to the v0.2.6 one (engine 0.1.51), I compared the two engines on the same server config with tools/ab_engine.py (greedy, 256 new tokens, alternating fresh engines; per round the tool's three chats of 27-34 tokens plus one 6,345-token prompt; then a 21,102-token prompt). Decode was unchanged. Prompt reading with the new Battlemage default (STRATA_PREFILL_MMQ unset = auto) was slower than the old engine, and its greedy answers changed from start to start.

6,345-token prompt, Qwen3.8-Flash-Next IQ3_S, --prefill 4096, one row per engine start:

engine / setting	starts	prompt tok/s	greedy answer length (tokens)
0.1.45 (FP16 only)	2	1318, 1307	151, 151
0.2.6 default (auto)	4	1107, 1115, 1116, 1118	134, 145, 134, 129
0.2.6 STRATA_PREFILL_MMQ=off (FP16 chain)	4	1316, 1316 median of 3	151 every time
0.2.6 STRATA_PREFILL_MMQ=1	2	1376, 1366	143, 143

21,102-token prompt: 0.1.45 1394, 1397; off 1400, 1405; =1 1431, 1431 (its answer differs from FP16 but is the same at both starts); =1 with --prefill 8192 1322, 1145, with decode afterwards 3-7 % lower. Decode after the 6.3k prompt: FP16 46.7-46.8, auto 46.2-47.6, =1 49.2 tok/s.

So on this card the auto default is the slowest of the three at these prompt sizes (15 % behind FP16 at 6.3k) and the only one whose greedy output is not repeatable. =1 is the fastest (+4.5 % at 6.3k, +2.5 % at 21k) and repeatable; the FP16 chain is byte-identical to 0.1.45, and again to itself on the v0.2.7.1 build (32 of 32 requests). I did not measure 1K or 4K prompts, where the 0.2.0 release notes measured the gain.

Why the answers vary: the crossover is measured at every start, and the measurement is not stable. The engine line over 11 starts on this card, same device string:

strata: int8 experts, measured here (Intel(R) Arc(TM) Pro B70 Graphics|1.17.39758+10|22|42|640x2560): us per expert at 16/32/64/128/256 rows, int8 17.5/30.5/36.0/49.8/83.4, FP16 29.7/31.8/36.4/47.5/67.2 -> crossover 72 rows
... int8 17.2/31.8/45.0/53.5/86.8, FP16 29.6/31.4/36.6/47.9/67.4 -> crossover 32 rows
... int8 17.2/28.1/44.0/52.0/84.2, FP16 28.6/30.6/36.6/48.1/66.8 -> crossover 40 rows
... int8 17.1/28.1/31.4/43.6/92.8, FP16 28.3/30.5/36.5/47.8/67.6 -> crossover 149 rows

Crossovers seen: 32, 40, 40, 40, 41, 72, 149, 151 rows (and 139, 151, 154 on three starts whose device string ends |22|20|640x2560). The FP16 timings are steady (36.4-36.6 us at 64 rows); the int8 timing at 64 rows swings between 31 and 45 us. Which groups take int8 therefore differs per start, and with it the rounding and the greedy answer.

The cache that should make this a one-time measurement (~/.cache/strata/i8-crossover.txt, STRATA_CACHE_DIR moves it, docs/INTEL_INT8_EXPERTS.md) is written inside the container that strata-sycl.sh starts with --rm -u $(id -u). That home is not a mounted volume (the only mounts are the repo's parent as /work and /dev/shm), so nothing persists and every start measures again.

The engine's advice line STRATA_PREFILL_MMQ=0 turns them off cannot be followed through the launcher, which drops a value of 0 (#4546); off works.

Suggestions, in the order I would try them: (1) let the launcher set STRATA_CACHE_DIR to a folder under /work so the crossover persists and starts are repeatable; (2) measure the crossover more robustly (repeats, median: the int8 timings at 64 rows are the unstable ones); (3) on this card either the FP16 chain or =1 would be a better default than auto at prompts of this size: both are repeatable and both were faster than auto here.

Smallest steps or command that reproduce it

1. Intel install, Qwen3.8-Flash-Next IQ3_S, the config setup writes (STRATA_PREFILL_MMQ unset).
2. Start the server three times; grep crossover in the engine log gives different numbers.
3. One ~6k-token prompt at temperature 0 per start: prompt_per_second about 1,110 and answers of different length. The same with "env": {"STRATA_PREFILL_MMQ": "off"}: about 1,315 and the same answer at every start.

Reproduction status

Reproduced locally

Strata version

Measured on the v0.2.6 source build (engine 0.1.51) on 2026-10-26; sycl/src/prefill/prefill.cpp, sycl/include/strata/prefill/esimd_moe.hpp and the launcher are unchanged in v0.2.7.1 (46c152a1) and on main today. On v0.2.7.1 (engine 0.1.53) I ran the FP16 chain only.

Graphics card, memory and system

Intel Arc Pro B70 32 GB · 62 GB RAM · Debian 13 (kernel 7.2.6, xe driver) · engine in the strata-sycl-dev container (oneAPI 2026.1; compute runtime 1.17.39758+10 as the engine prints it)

Model, size and settings

Qwen3.8-Flash-Next GSQ-RCO IQ3_S native pack, --stream-experts, --prefill 4096, --kv int8 --kv-resident 32768, 262144 context, --spec 4 with the MTP drafter, pictures off. Greedy, 256 new tokens.

Logs

The crossover lines above come from the engine logs of the auto starts; I can attach the full logs of the four auto starts and the ab_engine.py jsonl if useful.