prefill: ~7% pp512 regression on Arc B580 since #30160 (embeddings get_rows reorder), constant per-invocation cost

Open
  • Is something wrong?
#30,939 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp
Domain
performance

Research direction

Read commit #30160 and inspect the graph construction path that reorders get_rows for embeddings, focusing on the plain token-only path described in the report. Reproduce the pp512 benchmark on an Arc B580 with the stated llama-bench settings, comparing builds before and after #30160. Done means identifying whether the added per-invocation host-side cost is expected and whether the ordering can be limited to graphs that need it.

Written by the indexing model from the issue text.

Description

Setup: Intel Arc B580 (Xe2), driver 32.0.101.9150, Windows 11, MinGW Release build (static, Vulkan backend), llama-bench -t 14 -ub 512 -p {512,4096,16384} -n 128 -fa auto, gemma-4-E4B Q4_0 (dense, standard token path, no raw embeddings, no multimodal inputs), warm card.

Bisecting the prefill regression over 448147d42..b11756 (279 commits, stock builds, no local patches) isolates two independent commits: first bad is 5e4878e97 (#30145, tracked separately in #30381), second bad is 8e2d31e0e (#30160, "graph : reorder get_rows for embeddings").

pp512, stock builds:

build pp512
448147d42 (07.10) 3356 t/s
b11756 3102 t/s (-7.6%)

Bisect details:

  • phase 1 (trees as-is): first bad = 5e4878e97 (#30145) -> #30381
  • phase 2 (with 5e4878e97 locally reverted at every step, 230 commits): first bad = 8e2d31e0e (#30160). A plain git revert of it no longer applies on master (later commits build on top of it), so each bisect step was revert -> build -> bench -> reset.

Second confirmation on a local integration branch, interleaved A/B (4 pairs, binaries alternated within a single session; identical-binary control: mean -0.4%, single run +/- 2%; my branch carries a local FA capability-gate relaxation for B580, see the provenance note in #30018 - deltas are internally consistent, absolute values on stock builds differ):

tree pp512
pre-rebase (predates #30160) 4294 t/s
b11756 + local fixes + local revert of #30145 4019 t/s (-6.9%, 4/4 pairs)

Re-enabling #30145 on top of the same tree costs another -3.9% (3995 -> 3838 t/s), consistent with #30381.

Characteristics of the #30160 cost:

  • Prefill-only and confined to short prompts: pp512 -6.9% (the only statistically solid point: interleaved, +8.17 ms per invocation); pp4096 -1.2% and pp16384 ~0% are within single-shot noise (+/- 2%). Profile is consistent with a one-time per-invocation overhead (~8 ms), not a per-token slowdown.
  • Same delta with -fa off and -fa auto; per-op GPU time sums are unchanged across the boundary, so the cost is host-side (graph build / first-batch sync), not shader time.
  • decode (tg128) is not affected by this cost; across the whole commit range decode actually improved +5.7%, not attributable to this commit.

#30160 is a correctness fix (mixed/raw-embedding branches, gemma4 PLE, etc.), so I am not proposing a revert. Question: is the extra one-time cost on the plain token-only path expected? For a dense model with only token embeddings the reordered split-0 construction appears to add a host-side sync that did not exist before; if the new ordering could be limited to graphs that actually need it (mixed batch / raw embeddings), the standard path would keep the old timing.

Dominant language
C++
Stars
133k
Forks
25.1k
Avg merge
2d 11h
Merged PRs (30d)
505

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ggml-org/llama.cpp

All issues in ggml-org/llama.cpp

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.