Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Vulkan: batched encoder runs the rel-pos bd matmul on the CPU (ggml-vulkan rejects the ne3 broadcast)

Abierto
  • ¿Algo no está bien?
#101 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
47/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp
Área
backend, performance

Línea de trabajo

Start in RelPosAttention::build_graph_batched and build_graph_batched_local, where the issue identifies the shared positional tensor and batched matrix multiplication. Read the Vulkan MUL_MAT support check and mul_mm.comp’s dim-3 broadcast handling to understand the mismatch. Compare the proposed repeat_4d change against relaxing the check, then verify batched graph scheduling and the listed throughput and transcript results.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

With the Vulkan backend, RelPosAttention::build_graph_batched runs one matmul per encoder layer, and the rel-shift after it, on the CPU whenever B > 1. ph is [dk, P, H, 1] and broadcasts over the batch in ggml_mul_mat(ctx, ph, qv), but ggml_backend_vk_device_supports_op rejects GGML_OP_MUL_MAT when a->ne[3] != b->ne[3]. So Backend::compute sends every batched graph through the scheduler, with a CPU split in each of the 24 layers. On our machine the GPU sits around 30% busy and throughput follows the CPU thread count.

Setup

  • parakeet.cpp v0.6.1, parakeet-v0.6.1-lib-win-vulkan-x64. A source build of v0.6.1 (same flags as the release job) behaves identically.
  • Windows 11, AMD Radeon RX 6900 XT, AMD proprietary driver.
  • tdt-0.6b-v2-q8_0.gguf, the first 500 utterances of LibriSpeech test-clean, batched through the C API.

Evidence

GGML_SCHED_DEBUG=2 at batch 8: 100 splits per encoder graph, 50 of them on the CPU. Every layer has:

## SPLIT #2: CPU # 2 inputs: [ (reshaped) (permuted) (cont) (   1M)] [node_80 (   3M)]
node # 81 (   MUL_MAT):              node_81 (  15M) [  CPU         ]
node # 82 (       PAD):              node_82 (  15M) [  CPU         ]
node # 85 (      CONT):  (reshaped) (view) ( (  15M) [  CPU         ]
node # 88 (      CONT):  (reshaped) (view) ( (   7M) [  CPU         ]

The same ne[3] check is still in ggml master (a05c992, 2026-10-15), so updating ggml does not change this.

Patch we are running

@@ RelPosAttention::build_graph_batched
     ggml_tensor* ph = to_heads(p, pos_len);  // [dk, P, H]  (ne3=1, broadcast over B)
+    // ggml-vulkan's mul_mat requires equal ne3 on both operands, so a shared ph
+    // sends bd = ph @ qv (and the rel-shift after it) to the CPU when B > 1.
+    // Repeating ph over the batch keeps every layer on the GPU.
+    if (B > 1) ph = ggml_repeat_4d(ctx, ph, dk, pos_len, H, B); // [dk, P, H, B]
@@ RelPosAttention::build_graph_batched_local
     ggml_tensor* vh = to_heads_b(v, T), *php = to_heads(p, P);     // ph shared (ne3=1)
+    // Same as build_graph_batched: equal ne3 keeps bd on the GPU under Vulkan.
+    if (B > 1) php = ggml_repeat_4d(ctx, php, dk, P, H, B);         // [dk, P, H, B]

With it, every op in the batched graph is supported on Vulkan, so the graph takes the gallocr path and GGML_SCHED_DEBUG prints nothing.

Results

Same machine, the benchmark process pinned to 4 cores, 500 clips, real-time factor (audio seconds per wall second):

Batch v0.6.1 release Patched
2 74.6x 94.1x
4 97.6x 144.5x
8 112.5x 149.7x
12 39.2x 225.8x
16 39.4x 189.1x
24 40.8x 76.2x
  • Whole-machine CPU during the run: 58% before, 20% after.
  • WER 1.77 to 1.79% before, 1.79% after. Batched transcripts now match the B=1 transcripts on 499 of 500 clips (493 of 500 before; the CPU path's numerics differ slightly from the GPU's).
  • VRAM: about 400 MB more at batch 16, for the repeated ph.
  • 0 failures in any run.

Notes

  • mul_mm.comp already handles a dim-3 broadcast (i03 = i13 / p.broadcast3), so the ne[3] check in ggml_backend_vk_device_supports_op may be stricter than it needs to be. Relaxing it in ggml would make the repeat unnecessary. We have not tested that.
  • Probably unrelated, but visible only with the patch: once a batched graph has run through the persistent gallocr, later B=1 graphs run about twice as fast (24x to 53x on the same 100 clips) as when the gallocr was first created by a B=1 graph. With the release, batched graphs go through the scheduler, so B=1 stays at 24x. We have not looked into why.
  • Throughput still drops past batch 16 with the patch (76x at 24). That looks like a separate effect.
Lenguaje dominante
C++
Estrellas
844
Forks
103
Merge medio
8 h 27 min
PR fusionados (30 d)
26

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de mudler/parakeet.cpp

Todos los issues de mudler/parakeet.cpp

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.