Eval bug: Vulkan: count_experts.comp zero_skipped_rows() loops over every id in every thread

Closed Beginner friendly
  • Is something wrong?
#31,163 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@SG-Amadeus is already working on this.

Since Oct 27, 2026.

  • #31266 by @SG-Amadeus — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
backend, performance

Research direction

Read ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp at the linked zero_skipped_rows() loop and compare how the outer and inner loops distribute work by thread. The issue reports a tested loop change and provides benchmark commands and results; run the listed Vulkan backend tests with test-backend-ops -b Vulkan0. Done means the loop assigns expert ids to each thread as intended and the reported tests pass.

Written by the indexing model from the issue text.

Description

https://github.com/ggml-org/llama.cpp/blob/6050b467c07a95c7b87debd4ddac2f8ba8d9b807/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp#L59

In count_experts.comp, the zero_skipped_rows() loop (from #26631) runs over every expert id in every thread instead of each thread checking the ids specific to itself. Swapping the two loops speeds up prompt processing 5%-35% depending on the model . Generation isn't affected significantly since it's only one token.

Here's a fix that Opus 5.5 wrote and I tested:

diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp b/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp
index a46a0619..37767aec 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp
@@ -56,7 +56,7 @@ void zero_skipped_rows() {
     const uint num_elements = p.ne00 * p.ne01;
     const uint tid = gl_LocalInvocationID.x;

-    for (uint idx = 0; idx < num_elements; idx++) {
+    for (uint idx = tid; idx < num_elements; idx += BLOCK_SIZE) {
         const uint i01 = fastdiv(idx, p.ne00mp, p.ne00L);
         const uint i00 = idx - i01 * p.ne00;

@@ -65,7 +65,7 @@ void zero_skipped_rows() {
         }

         const uint d_off = i00 * p.stride_dst + i01 * p.batch_stride_dst;
-        for (uint i = tid; i < p.ne0_dst; i += BLOCK_SIZE) {
+        for (uint i = 0; i < p.ne0_dst; i++) {
             data_dst[d_off + i] = 0.0f;
         }
     }
model experts (used/total) pp2048 master pp2048 patched change tg128 master tg128 patched
granite-3.1-3b-a800m Q4_K_M 8/40 6399.06 8625.55 +35% 316.36 316.48
DeepSeek-V2-Lite-Chat 16B Q4_K_M 6/64 3976.99 4369.85 +10% 188.27 187.15
gpt-oss-20b MXFP4 4/32 3297.97 3511.33 +6% 161.61 161.59
Qwen3.6-35B-A3B Q4_K_XL (-ncmoe 24) GPU w/ CPU 8/256 754.48 791.08 +5% 28.14 27.77
Qwen3-1.7B Q8_0 (dense, control) - 8330.22 8283.40 -1% 221.37 220.96

RX 7800 XT, RADV (Mesa 25.2.8), Linux 6.8, measured on master 2280f0fd

llama-bench -m granite-3.1-3b-a800m-instruct-Q4_K_M.gguf -m gpt-oss-20b-MXFP4.gguf -ngl 99 -fa 1 -lm none -ub 2048 -p 512,2048 -n 128 -r 3
llama-bench -m DeepSeek-V2-Lite-Chat-Q4_K_M.gguf -ngl 99 -fa 1 -lm none -ub 2048 -p 512,2048 -n 128 -r 3
llama-bench -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -ngl 99 -ncmoe 24 -fa 1 -ub 2048 -p 512,2048 -n 64 -r 3
llama-bench -m Qwen3-1.7B-Q8_0.gguf -ngl 99 -fa 1 -lm none -ub 2048 -p 512,2048 -n 128 -r 5

test-backend-ops -b Vulkan0: all 21303 tests pass with the patch (including MUL_MAT_ID 1109/1109 and its 170 skip cases).

Dominant language
C++
Stars
133k
Forks
25.1k
Avg merge
2d 11h
Merged PRs (30d)
505

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ggml-org/llama.cpp

All issues in ggml-org/llama.cpp

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.