Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Vulkan: batched encoder runs the rel-pos bd matmul on the CPU (ggml-vulkan rejects the ne3 broadcast)

Aperta
  • Qualcosa non va?
#101 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
47/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, performance

Direzione di ricerca

Start in RelPosAttention::build_graph_batched and build_graph_batched_local, where the issue identifies the shared positional tensor and batched matrix multiplication. Read the Vulkan MUL_MAT support check and mul_mm.comp’s dim-3 broadcast handling to understand the mismatch. Compare the proposed repeat_4d change against relaxing the check, then verify batched graph scheduling and the listed throughput and transcript results.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

With the Vulkan backend, RelPosAttention::build_graph_batched runs one matmul per encoder layer, and the rel-shift after it, on the CPU whenever B > 1. ph is [dk, P, H, 1] and broadcasts over the batch in ggml_mul_mat(ctx, ph, qv), but ggml_backend_vk_device_supports_op rejects GGML_OP_MUL_MAT when a->ne[3] != b->ne[3]. So Backend::compute sends every batched graph through the scheduler, with a CPU split in each of the 24 layers. On our machine the GPU sits around 30% busy and throughput follows the CPU thread count.

Setup

  • parakeet.cpp v0.6.1, parakeet-v0.6.1-lib-win-vulkan-x64. A source build of v0.6.1 (same flags as the release job) behaves identically.
  • Windows 11, AMD Radeon RX 6900 XT, AMD proprietary driver.
  • tdt-0.6b-v2-q8_0.gguf, the first 500 utterances of LibriSpeech test-clean, batched through the C API.

Evidence

GGML_SCHED_DEBUG=2 at batch 8: 100 splits per encoder graph, 50 of them on the CPU. Every layer has:

## SPLIT #2: CPU # 2 inputs: [ (reshaped) (permuted) (cont) (   1M)] [node_80 (   3M)]
node # 81 (   MUL_MAT):              node_81 (  15M) [  CPU         ]
node # 82 (       PAD):              node_82 (  15M) [  CPU         ]
node # 85 (      CONT):  (reshaped) (view) ( (  15M) [  CPU         ]
node # 88 (      CONT):  (reshaped) (view) ( (   7M) [  CPU         ]

The same ne[3] check is still in ggml master (a05c992, 2026-10-15), so updating ggml does not change this.

Patch we are running

@@ RelPosAttention::build_graph_batched
     ggml_tensor* ph = to_heads(p, pos_len);  // [dk, P, H]  (ne3=1, broadcast over B)
+    // ggml-vulkan's mul_mat requires equal ne3 on both operands, so a shared ph
+    // sends bd = ph @ qv (and the rel-shift after it) to the CPU when B > 1.
+    // Repeating ph over the batch keeps every layer on the GPU.
+    if (B > 1) ph = ggml_repeat_4d(ctx, ph, dk, pos_len, H, B); // [dk, P, H, B]
@@ RelPosAttention::build_graph_batched_local
     ggml_tensor* vh = to_heads_b(v, T), *php = to_heads(p, P);     // ph shared (ne3=1)
+    // Same as build_graph_batched: equal ne3 keeps bd on the GPU under Vulkan.
+    if (B > 1) php = ggml_repeat_4d(ctx, php, dk, P, H, B);         // [dk, P, H, B]

With it, every op in the batched graph is supported on Vulkan, so the graph takes the gallocr path and GGML_SCHED_DEBUG prints nothing.

Results

Same machine, the benchmark process pinned to 4 cores, 500 clips, real-time factor (audio seconds per wall second):

Batch v0.6.1 release Patched
2 74.6x 94.1x
4 97.6x 144.5x
8 112.5x 149.7x
12 39.2x 225.8x
16 39.4x 189.1x
24 40.8x 76.2x
  • Whole-machine CPU during the run: 58% before, 20% after.
  • WER 1.77 to 1.79% before, 1.79% after. Batched transcripts now match the B=1 transcripts on 499 of 500 clips (493 of 500 before; the CPU path's numerics differ slightly from the GPU's).
  • VRAM: about 400 MB more at batch 16, for the repeated ph.
  • 0 failures in any run.

Notes

  • mul_mm.comp already handles a dim-3 broadcast (i03 = i13 / p.broadcast3), so the ne[3] check in ggml_backend_vk_device_supports_op may be stricter than it needs to be. Relaxing it in ggml would make the repeat unnecessary. We have not tested that.
  • Probably unrelated, but visible only with the patch: once a batched graph has run through the persistent gallocr, later B=1 graphs run about twice as fast (24x to 53x on the same 100 clips) as when the gallocr was first created by a B=1 graph. With the release, batched graphs go through the scheduler, so B=1 stays at 24x. We have not looked into why.
  • Throughput still drops past batch 16 with the patch (76x at 24). That looks like a separate effect.
Lingua principale
C++
Stelle
844
Fork
103
Merge medio
8h 27m
PR unite (30g)
26

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di mudler/parakeet.cpp

Tutte le issue di mudler/parakeet.cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.