v3d: "Resetting GPU for hang" during a compute job that is making progress drops its last workgroups, with no error to userspace
Mantenedores costumam responder em até 1 dia
Ninguém assumiu esta issue ainda.
Avaliação
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Facilidade para iniciantes
- 12/100
- Tipo de issue
- Bug
- Clareza
- Razoavelmente clara
- Status de atividade
- Ativa
- Stack de tecnologia
- c, linux
- Domínio
- computer-graphics, operating-systems
Direção de pesquisa
Read the CSD timeout path in drivers/gpu/drm/v3d/v3d_sched.c: the handler that samples V3D_CSD_CURRENT_CFG4 and re-arms the timer, plus v3d_reset and how the fence is signalled. It needs a Pi 5 with Mesa V3DV and the reporter's locus repro to observe, and no standalone repro exists, so the first step is trying to build a minimal hang-triggering submit. Done means a job still making progress is not reset, or userspace learns the work was lost.
Escrita pelo modelo de indexação a partir do texto da issue.
Descrição
Describe the bug
On a Raspberry Pi 5, a long Vulkan compute dispatch sometimes triggers the v3d driver's hang check. The GPU is reset and the job's remaining workgroups never run. The command buffer's fence still signals, and no error reaches Vulkan (no VK_ERROR_DEVICE_LOST), so the application reads a result with a block of unwritten outputs. The job that tripped it ran 274 ms in total, under the 500 ms scheduler timeout, and was making progress.
The code involved is drivers/gpu/drm/v3d (the CSD timeout handling in v3d_sched.c), so this may also apply to the upstream drm/v3d driver; we have only run the Raspberry Pi kernel.
The dispatch that hit it: a compute shader (an implicit-GEMM 3 x 3 convolution, 256 invocations per workgroup, 12 KiB of shared memory), a grid of 338 x 1 x 1 workgroups, 274 ms of GPU time for the whole dispatch (timestamp queries), one dispatch per command buffer, one submit and a fence wait each.
- With the output buffer filled with NaN before the dispatch, about 13700 of its 691488 outputs were still NaN afterwards. They are exactly the outputs of the last 6 or 7 of the 338 workgroups (output pixels 21248 to 21608, in every channel). A second read 20 ms later is the same, so these are not late writes: those workgroups never ran.
- A shader-write to host-read memory barrier at the end of the command buffer does not change it.
- It happened in about 1 run in 30 of that dispatch, when other dispatches of other sizes ran between runs; the same dispatch repeated on its own 2500 times never triggered it.
- One kernel reset per wrong result: in one test of 300 rounds, 11 resets in the kernel log and 11 wrong results. Over full test suites of the application (many long compute jobs), 43 and 24 resets in two runs.
Steps to reproduce the behaviour
We have no standalone repro yet; the repro below needs our application (open source, https://github.com/garana/locus), whose test suite has the case as a hidden test. On a Pi 5 with the system Mesa:
git clone https://github.com/garana/locus && cd locus
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j4
# LOCUS_VK_SPLIT=0 turns off the application's workaround.
LOCUS_VK_SPLIT=0 LOCUS_VK_STRESS_ITERS=300 \
./build/tests/locus_tests "vulkan conv2d repeats its bits"
journalctl -k | grep "Resetting GPU"
The case runs five convolution dispatches of different sizes in turn, 300 rounds, on fixed inputs, and reports every round whose output differs from the first one's. One run gave 15 wrong rounds and 15 resets in the kernel log; with the workaround on (no LOCUS_VK_SPLIT), 0 and 0. A wrong round prints like:
conv 32 -> 32 k 3 at 149: round 33: 2243 of 691488 outputs differ, channels 0 .. 4, pixels 21248 .. 21608
Standalone attempts did not trigger it: a 338-workgroup dispatch of a pure ALU loop, and one with 12 KiB of shared memory and barriers in the loop, both running about 280 ms, gave 0 resets in 600 rounds each, with and without a short dispatch between rounds. The convolution kernel also streams its input and weights from memory (about 3 MB a dispatch), so memory traffic may be part of the condition.
Device (s)
Raspberry Pi 5
System
- Raspberry Pi 5 Model B Rev 1.0, 4 GB; V3D 7.1.7.0
- OS: Raspberry Pi reference 2024-03-15 (Debian 12, bookworm), aarch64
- Firmware: 2025/12/08 19:29:54, version 2226a853 (release) (embedded)
- Kernel: 6.12.75+rpt-rpi-2712, #1 SMP PREEMPT Debian 1:6.12.75-1+rpt1~bookworm (2026-03-11)
- Mesa 24.2.8-1~bpo12+rpt4 (V3DV), Vulkan 1.2.289
Logs
Once per dropped job:
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
Additional context
Workaround: recording the same grid as several vkCmdDispatchBase calls of at most 32 workgroups each (v3dv runs each as its own CSD job, so every job is short) avoids it completely: 0 resets and 0 wrong results in the same 300 rounds and over a full test suite, with no measurable change in total time.
Our reading of the cause (unconfirmed): as we read drivers/gpu/drm/v3d/v3d_sched.c, the CSD timeout handler re-arms the timer only while V3D_CSD_CURRENT_CFG4 (the batches remaining) changes between two checks. Near the end of a long job, with the last batches already dispatched and still running, that counter may stop changing for one timer period although the job is progressing, and the job is then taken for hung. If so, the last batches in flight are lost at the reset.
The measurements are in https://github.com/garana/locus/issues/682.
- Linguagem predominante
- C
- Estrelas
- 13.3k
- Forks
- 5.5k
- Merge médio
- 4d 10h
- PRs com merge (30d)
- 31
Preparar o ambiente
Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.
Primeiros passos
- Leia a issue inteira e depois o guia de contribuição do projeto.
- Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
- Faça um fork do repositório e trabalhe em uma branch.
- Abra um pull request que referencie o número da issue.
Mais de raspberrypi/linux
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 68/100
raspberrypi/linux#7415 · 2 comentários · 1 reação ·
Mantenedores costumam responder em até 1 dia
-
rp1-cfe doesn't forward V4L2_EVENT_SOURCE_CHANGE event from csi-2 sensor driver to userspace appAberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 72/100
raspberrypi/linux#7399 · 1 comentário ·
Mantenedores costumam responder em até 1 dia
-
Dificuldade 1/5 Menos de uma hora Facilidade para iniciantes 82/100
raspberrypi/linux#7357 · 1 comentário ·
Mantenedores costumam responder em até 1 dia
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 68/100
raspberrypi/linux#7054 · 2 comentários ·
Mantenedores costumam responder em até 1 dia
-
Dificuldade 4/5 3-5 dias Facilidade para iniciantes 48/100
raspberrypi/linux#7770 ·
Mantenedores costumam responder em até 1 dia
Todas as issues de raspberrypi/linux
Issues semelhantes
-
Dificuldade 1/5 Menos de uma hora Facilidade para iniciantes 82/100
LedgerHQ/app-bitcoin-legacy#306 ·
-
[issue]: Ventoy2Disk.sh can leave the end of VTOYEFI unwritten (xzcat | dd without iflag=fullblock)Aberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 78/100
-
bug
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 73/100
prusa3d/Prusa-Firmware-Buddy#5629 ·
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 84/100
Mantenedores costumam responder em até 3 dias
-
Issue PRL on WindowsAberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 78/100
Mantenedores costumam responder em até 1 dia