v3d: "Resetting GPU for hang" during a compute job that is making progress drops its last workgroups, with no error to userspace
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 12/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- c, linux
Línea de trabajo
Read the CSD timeout path in drivers/gpu/drm/v3d/v3d_sched.c: the handler that samples V3D_CSD_CURRENT_CFG4 and re-arms the timer, plus v3d_reset and how the fence is signalled. It needs a Pi 5 with Mesa V3DV and the reporter's locus repro to observe, and no standalone repro exists, so the first step is trying to build a minimal hang-triggering submit. Done means a job still making progress is not reset, or userspace learns the work was lost.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Describe the bug
On a Raspberry Pi 5, a long Vulkan compute dispatch sometimes triggers the v3d driver's hang check. The GPU is reset and the job's remaining workgroups never run. The command buffer's fence still signals, and no error reaches Vulkan (no VK_ERROR_DEVICE_LOST), so the application reads a result with a block of unwritten outputs. The job that tripped it ran 274 ms in total, under the 500 ms scheduler timeout, and was making progress.
The code involved is drivers/gpu/drm/v3d (the CSD timeout handling in v3d_sched.c), so this may also apply to the upstream drm/v3d driver; we have only run the Raspberry Pi kernel.
The dispatch that hit it: a compute shader (an implicit-GEMM 3 x 3 convolution, 256 invocations per workgroup, 12 KiB of shared memory), a grid of 338 x 1 x 1 workgroups, 274 ms of GPU time for the whole dispatch (timestamp queries), one dispatch per command buffer, one submit and a fence wait each.
- With the output buffer filled with NaN before the dispatch, about 13700 of its 691488 outputs were still NaN afterwards. They are exactly the outputs of the last 6 or 7 of the 338 workgroups (output pixels 21248 to 21608, in every channel). A second read 20 ms later is the same, so these are not late writes: those workgroups never ran.
- A shader-write to host-read memory barrier at the end of the command buffer does not change it.
- It happened in about 1 run in 30 of that dispatch, when other dispatches of other sizes ran between runs; the same dispatch repeated on its own 2500 times never triggered it.
- One kernel reset per wrong result: in one test of 300 rounds, 11 resets in the kernel log and 11 wrong results. Over full test suites of the application (many long compute jobs), 43 and 24 resets in two runs.
Steps to reproduce the behaviour
We have no standalone repro yet; the repro below needs our application (open source, https://github.com/garana/locus), whose test suite has the case as a hidden test. On a Pi 5 with the system Mesa:
git clone https://github.com/garana/locus && cd locus
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j4
# LOCUS_VK_SPLIT=0 turns off the application's workaround.
LOCUS_VK_SPLIT=0 LOCUS_VK_STRESS_ITERS=300 \
./build/tests/locus_tests "vulkan conv2d repeats its bits"
journalctl -k | grep "Resetting GPU"
The case runs five convolution dispatches of different sizes in turn, 300 rounds, on fixed inputs, and reports every round whose output differs from the first one's. One run gave 15 wrong rounds and 15 resets in the kernel log; with the workaround on (no LOCUS_VK_SPLIT), 0 and 0. A wrong round prints like:
conv 32 -> 32 k 3 at 149: round 33: 2243 of 691488 outputs differ, channels 0 .. 4, pixels 21248 .. 21608
Standalone attempts did not trigger it: a 338-workgroup dispatch of a pure ALU loop, and one with 12 KiB of shared memory and barriers in the loop, both running about 280 ms, gave 0 resets in 600 rounds each, with and without a short dispatch between rounds. The convolution kernel also streams its input and weights from memory (about 3 MB a dispatch), so memory traffic may be part of the condition.
Device (s)
Raspberry Pi 5
System
- Raspberry Pi 5 Model B Rev 1.0, 4 GB; V3D 7.1.7.0
- OS: Raspberry Pi reference 2024-03-15 (Debian 12, bookworm), aarch64
- Firmware: 2025/12/08 19:29:54, version 2226a853 (release) (embedded)
- Kernel: 6.12.75+rpt-rpi-2712, #1 SMP PREEMPT Debian 1:6.12.75-1+rpt1~bookworm (2026-03-11)
- Mesa 24.2.8-1~bpo12+rpt4 (V3DV), Vulkan 1.2.289
Logs
Once per dropped job:
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
Additional context
Workaround: recording the same grid as several vkCmdDispatchBase calls of at most 32 workgroups each (v3dv runs each as its own CSD job, so every job is short) avoids it completely: 0 resets and 0 wrong results in the same 300 rounds and over a full test suite, with no measurable change in total time.
Our reading of the cause (unconfirmed): as we read drivers/gpu/drm/v3d/v3d_sched.c, the CSD timeout handler re-arms the timer only while V3D_CSD_CURRENT_CFG4 (the batches remaining) changes between two checks. Near the end of a long job, with the last batches already dispatched and still running, that counter may stop changing for one timer period although the job is progressing, and the job is then taken for hung. If so, the last batches in flight are lost at the reset.
The measurements are in https://github.com/garana/locus/issues/682.
- Lenguaje dominante
- C
- Estrellas
- 13.3k
- Forks
- 5.5k
- Merge medio
- 4 d 10 h
- PR fusionados (30 d)
- 31
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de raspberrypi/linux
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
raspberrypi/linux#7415 · 2 comentarios · 1 reacción ·
Los mantenedores suelen responder en 1 día
-
rp1-cfe doesn't forward V4L2_EVENT_SOURCE_CHANGE event from csi-2 sensor driver to userspace appAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
raspberrypi/linux#7399 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Documentation update requiredAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 82/100
raspberrypi/linux#7357 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
raspberrypi/linux#7054 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
Dificultad 4/5 3-5 días Aptitud para principiantes 48/100
raspberrypi/linux#7770 ·
Los mantenedores suelen responder en 1 día
Todos los issues de raspberrypi/linux
Issues similares
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 82/100
LedgerHQ/app-bitcoin-legacy#306 ·
-
[issue]: Ventoy2Disk.sh can leave the end of VTOYEFI unwritten (xzcat | dd without iflag=fullblock)Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 73/100
prusa3d/Prusa-Firmware-Buddy#5629 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
Los mantenedores suelen responder en 3 días
-
Issue PRL on WindowsAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día