v3d: "Resetting GPU for hang" during a compute job that is making progress drops its last workgroups, with no error to userspace

Aperta
#7,759 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, linux

Direzione di ricerca

Read the CSD timeout path in drivers/gpu/drm/v3d/v3d_sched.c: the handler that samples V3D_CSD_CURRENT_CFG4 and re-arms the timer, plus v3d_reset and how the fence is signalled. It needs a Pi 5 with Mesa V3DV and the reporter's locus repro to observe, and no standalone repro exists, so the first step is trying to build a minimal hang-triggering submit. Done means a job still making progress is not reset, or userspace learns the work was lost.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

On a Raspberry Pi 5, a long Vulkan compute dispatch sometimes triggers the v3d driver's hang check. The GPU is reset and the job's remaining workgroups never run. The command buffer's fence still signals, and no error reaches Vulkan (no VK_ERROR_DEVICE_LOST), so the application reads a result with a block of unwritten outputs. The job that tripped it ran 274 ms in total, under the 500 ms scheduler timeout, and was making progress.

The code involved is drivers/gpu/drm/v3d (the CSD timeout handling in v3d_sched.c), so this may also apply to the upstream drm/v3d driver; we have only run the Raspberry Pi kernel.

The dispatch that hit it: a compute shader (an implicit-GEMM 3 x 3 convolution, 256 invocations per workgroup, 12 KiB of shared memory), a grid of 338 x 1 x 1 workgroups, 274 ms of GPU time for the whole dispatch (timestamp queries), one dispatch per command buffer, one submit and a fence wait each.

  • With the output buffer filled with NaN before the dispatch, about 13700 of its 691488 outputs were still NaN afterwards. They are exactly the outputs of the last 6 or 7 of the 338 workgroups (output pixels 21248 to 21608, in every channel). A second read 20 ms later is the same, so these are not late writes: those workgroups never ran.
  • A shader-write to host-read memory barrier at the end of the command buffer does not change it.
  • It happened in about 1 run in 30 of that dispatch, when other dispatches of other sizes ran between runs; the same dispatch repeated on its own 2500 times never triggered it.
  • One kernel reset per wrong result: in one test of 300 rounds, 11 resets in the kernel log and 11 wrong results. Over full test suites of the application (many long compute jobs), 43 and 24 resets in two runs.
Steps to reproduce the behaviour

We have no standalone repro yet; the repro below needs our application (open source, https://github.com/garana/locus), whose test suite has the case as a hidden test. On a Pi 5 with the system Mesa:

git clone https://github.com/garana/locus && cd locus
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j4
# LOCUS_VK_SPLIT=0 turns off the application's workaround.
LOCUS_VK_SPLIT=0 LOCUS_VK_STRESS_ITERS=300 \
    ./build/tests/locus_tests "vulkan conv2d repeats its bits"
journalctl -k | grep "Resetting GPU"

The case runs five convolution dispatches of different sizes in turn, 300 rounds, on fixed inputs, and reports every round whose output differs from the first one's. One run gave 15 wrong rounds and 15 resets in the kernel log; with the workaround on (no LOCUS_VK_SPLIT), 0 and 0. A wrong round prints like:

conv 32 -> 32 k 3 at 149: round 33: 2243 of 691488 outputs differ, channels 0 .. 4, pixels 21248 .. 21608

Standalone attempts did not trigger it: a 338-workgroup dispatch of a pure ALU loop, and one with 12 KiB of shared memory and barriers in the loop, both running about 280 ms, gave 0 resets in 600 rounds each, with and without a short dispatch between rounds. The convolution kernel also streams its input and weights from memory (about 3 MB a dispatch), so memory traffic may be part of the condition.

Device (s)

Raspberry Pi 5

System
  • Raspberry Pi 5 Model B Rev 1.0, 4 GB; V3D 7.1.7.0
  • OS: Raspberry Pi reference 2024-03-15 (Debian 12, bookworm), aarch64
  • Firmware: 2025/12/08 19:29:54, version 2226a853 (release) (embedded)
  • Kernel: 6.12.75+rpt-rpi-2712, #1 SMP PREEMPT Debian 1:6.12.75-1+rpt1~bookworm (2026-03-11)
  • Mesa 24.2.8-1~bpo12+rpt4 (V3DV), Vulkan 1.2.289
Logs

Once per dropped job:

v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
Additional context

Workaround: recording the same grid as several vkCmdDispatchBase calls of at most 32 workgroups each (v3dv runs each as its own CSD job, so every job is short) avoids it completely: 0 resets and 0 wrong results in the same 300 rounds and over a full test suite, with no measurable change in total time.

Our reading of the cause (unconfirmed): as we read drivers/gpu/drm/v3d/v3d_sched.c, the CSD timeout handler re-arms the timer only while V3D_CSD_CURRENT_CFG4 (the batches remaining) changes between two checks. Near the end of a long job, with the last batches already dispatched and still running, that counter may stop changing for one timer period although the job is progressing, and the job is then taken for hung. If so, the last batches in flight are lost at the reset.

The measurements are in https://github.com/garana/locus/issues/682.

Lingua principale
C
Stelle
13.3k
Fork
5.5k
Merge medio
4g 10h
PR unite (30g)
31

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di raspberrypi/linux

Tutte le issue di raspberrypi/linux

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.