Eval bug: SYCL: XMX-ESIMD path makes MoE prompt processing ~35% slower on Arc Pro B70

Open
  • Is something wrong?
#31,068 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, linux
Domain
backend, performance

Research direction

Start with ggml_sycl_xmx_enabled() and the dispatch paths named in mmvq.cpp and ggml-sycl.cpp; compare their behavior with GGML_SYCL_ENABLE_XMX_ESIMD enabled and disabled for MoE prompt batches. The issue reports repeatable prefill timings and provides a reproduction using llama-server with a 32,799-token prompt. Done means identifying the affected dispatch or kernel behavior and resolving the reported prefill regression while checking the decode measurements.

Written by the indexing model from the issue text.

Description

SYCL: XMX-ESIMD path is ~35% slower for MoE prompt processing on Arc Pro B70 (regression in d81d0d8fb)

Summary

On an Intel Arc Pro B70 (Battlemage), the SYCL backend regressed prompt processing by
28–35% between 339f3242c (build 11783) and edc353842 (build 11851) for a qwen35moe
MoE model (Ornith-1.5-35B-A3B Q5_K_M, 256K ctx). Decode is unchanged.

The regression is fully attributable to the XMX-ESIMD path: running the same edc353842
binary with GGML_SYCL_ENABLE_XMX_ESIMD=0 restores the old numbers exactly (1,749 vs 1,131
t/s at 32K prompt, 1,597 vs 1,053 at 64K). The suspected commit is
d81d0d8fb95e52a66992bcfc212b1891b4d2f94a ("Multi-column on Intel XMX matrix engines
refactored", #29864), which is the only commit in the range that changes these kernels.

MoE inference is hit hard because expert routing leaves only a handful of tokens per expert
even during prefill, so the multi-column (2–80 columns) path that was written for
speculative-decoding verification also handles most of prompt processing.

Environment
GPU Intel(R) Arc(TM) Pro B70 Graphics, 32 GB (Xe2 / Battlemage), Level-Zero 1.17.39758, driver 26.35.39758
CPU / OS i7-14700K, Arch Linux, kernel 7.2.11-arch1-1
oneAPI 2026.0 (IntelLLVM 2026.0.0), oneDNN 3.11, MKL 2026.0, TBB 2023.0
Build -DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DMKL_LINK=dynamic -DMKL_SYCL_LINK=dynamic, icx/icpx
Old build 0.7.0-dev (build 11783, commit 339f3242c)
New build 0.7.0-dev (build 11851, commit edc353842)

Both binaries link libdnnl.so.3 and libmkl_sycl_blas.so.6 (verified with ldd); the local
tree only carries a dormant patch that replaces two GGML_ABORTs in the device-memory query
with a 30 GiB fallback, and it never fires on this box (the real query succeeds).

Model and flags

Ornith-1.5-35B-A3B Q5_K_M (qwen35moe, 24 GB, 256 experts, ~3B active/token), served by
llama-server with:

--gpu-layers 999 --fit off --ctx-size 262144
--flash-attn on --batch-size 2048 --ubatch-size 2048
--cache-type-k q8_0 --cache-type-v q8_0
--parallel 1 --cont-batching --metrics
--spec-type draft-mtp --spec-draft-n-max 3
--cache-ram 24576

Single slot, one request at a time, nothing else on the GPU. Env:
UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=0, SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1,
ZES_ENABLE_SYSMAN=1.

Measurements

Method: POST /completion with "cache_prompt": false and "n_predict": 64, one prompt per
size, reading timings.prompt_per_second and timings.predicted_per_second. Prompt sizes are
the tokenized lengths. The 32K point is measured twice; within-run repeatability is ~0.3% on
prefill and ~2% on decode.

Cross-build curve (same box, same flags, same day)
prompt tokens prefill b11783 prefill b11851 delta decode b11783 decode b11851
1,055 558.5 564.3 +1.0% 55.77 69.94
8,223 1,666.0 1,116.7 -33.0% 66.41 63.40
32,799 1,752.0 1,130.6 -35.5% 52.37 53.45
65,567 1,597.0 1,050.6 -34.2% 46.85 47.19
131,103 1,375.7 950.0 -30.9% 36.29 36.45
193,567 1,218.6 871.8 -28.4% 36.46 32.91
32,799 (repeat) 1,746.5 1,129.9 -35.3% 51.25 60.99

A rollback control (same b11783 binary, new process) reproduced 1,749.2 / 1,596.8 t/s at
32K / 64K, so the gap is not drift between runs.

Same-binary isolation (edc353842, only the env var differs)
flag 32K prefill 64K prefill 32K decode
GGML_SYCL_ENABLE_XMX_ESIMD=1 (default) 1,131.3 1,052.6 56.9
GGML_SYCL_ENABLE_XMX_ESIMD=0 1,748.9 1,597.3 58.1

Both runs report 0.7.0-dev (build 11851, commit edc353842); the flag was confirmed in
/proc/<pid>/environ at run time. The =0 numbers are within 0.1% of the b11783 build.

Reproduction

GGML_SYCL_ENABLE_XMX_ESIMD=1 llama-server -m model.gguf --gpu-layers 999 \
  --ctx-size 262144 --batch-size 2048 --ubatch-size 2048 \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --parallel 1 --metrics
# prompt 32799 tokens, cache_prompt=false  -> ~1,130 t/s

GGML_SYCL_ENABLE_XMX_ESIMD=0 llama-server ...   # same everything else
# prompt 32799 tokens, cache_prompt=false  -> ~1,749 t/s

Analysis / hypothesis

GGML_SYCL_ENABLE_XMX_ESIMD gates ggml_sycl_xmx_enabled(), which is consumed by:

  • mmvq.cpp — ggml_sycl_mul_mat_vec_q_xmx_type() (multi-column mmvq, 2..GGML_SYCL_XMX_MAX_COLS=80
    columns, plus the single-column Q2_K/Q3_K/Q8_0 case) and ggml_sycl_mul_mat_vec_q_glu_xmx();
  • ggml-sycl.cpp — can_use_xmx_batch(), which extends can_use_mul_mat_vec_q() past
    MMVQ_MAX_BATCH_SIZE.

None of those obviously matches a 2,048-token prefill batch, which is why I suspect the MoE
expert dispatch: after routing, each expert sees only a few tokens (roughly 2048/256 ≈ 8 on
average), which lands squarely in the 2–80 column range these kernels target. If that is the
case, the multi-column kernels are not just a spec-decode optimization for MoE models — they
sit on the prompt-processing hot path.

I have not profiled the kernels, so I cannot say whether the loss is occupancy, the reorder
layout, the 64-byte-alignment fallbacks, or the short-row variant; the numbers are stable and
reproducible either way.

What I am asking

  1. Is prompt processing on MoE models expected to route through the multi-column XMX path?
    If yes, this is a real regression for MoE prefill, not just a spec-decode trade-off.
  2. Is there a supported way to keep the decode gain while avoiding the prefill loss — e.g. a
    shape/column-range gate (GGML_SYCL_XMX_MAX_COLS, GGML_SYCL_XMX_GATHER_SHAPES), or should
    the dispatch avoid this path for large src1->ne[1]?
  3. Happy to test patches or run GGML_SYCL_DEBUG=1 with per-op timing if that helps.

For now the machine stays on 339f3242c; edc353842 is parked and the workaround is
GGML_SYCL_ENABLE_XMX_ESIMD=0, which merely ties the old build (no reason to upgrade with it).

Artifacts

Raw curves (JSON + markdown) from the four runs are available on request; the numbers above are
copied from them verbatim.

Related: #31033 is a compile failure introduced by the same commit on another Xe arch; this
report is the runtime performance side of the same change.

Dominant language
C++
Stars
133k
Forks
25.1k
Avg merge
2d 8h
Merged PRs (30d)
495

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ggml-org/llama.cpp

All issues in ggml-org/llama.cpp

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.