v3d: "Resetting GPU for hang" during a compute job that is making progress drops its last workgroups, with no error to userspace
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 12/100
- issue の種類
- バグ
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- c, linux
調査の方向性
Read the CSD timeout path in drivers/gpu/drm/v3d/v3d_sched.c: the handler that samples V3D_CSD_CURRENT_CFG4 and re-arms the timer, plus v3d_reset and how the fence is signalled. It needs a Pi 5 with Mesa V3DV and the reporter's locus repro to observe, and no standalone repro exists, so the first step is trying to build a minimal hang-triggering submit. Done means a job still making progress is not reset, or userspace learns the work was lost.
索引モデルが issue の本文から書いたものです。
説明
Describe the bug
On a Raspberry Pi 5, a long Vulkan compute dispatch sometimes triggers the v3d driver's hang check. The GPU is reset and the job's remaining workgroups never run. The command buffer's fence still signals, and no error reaches Vulkan (no VK_ERROR_DEVICE_LOST), so the application reads a result with a block of unwritten outputs. The job that tripped it ran 274 ms in total, under the 500 ms scheduler timeout, and was making progress.
The code involved is drivers/gpu/drm/v3d (the CSD timeout handling in v3d_sched.c), so this may also apply to the upstream drm/v3d driver; we have only run the Raspberry Pi kernel.
The dispatch that hit it: a compute shader (an implicit-GEMM 3 x 3 convolution, 256 invocations per workgroup, 12 KiB of shared memory), a grid of 338 x 1 x 1 workgroups, 274 ms of GPU time for the whole dispatch (timestamp queries), one dispatch per command buffer, one submit and a fence wait each.
- With the output buffer filled with NaN before the dispatch, about 13700 of its 691488 outputs were still NaN afterwards. They are exactly the outputs of the last 6 or 7 of the 338 workgroups (output pixels 21248 to 21608, in every channel). A second read 20 ms later is the same, so these are not late writes: those workgroups never ran.
- A shader-write to host-read memory barrier at the end of the command buffer does not change it.
- It happened in about 1 run in 30 of that dispatch, when other dispatches of other sizes ran between runs; the same dispatch repeated on its own 2500 times never triggered it.
- One kernel reset per wrong result: in one test of 300 rounds, 11 resets in the kernel log and 11 wrong results. Over full test suites of the application (many long compute jobs), 43 and 24 resets in two runs.
Steps to reproduce the behaviour
We have no standalone repro yet; the repro below needs our application (open source, https://github.com/garana/locus), whose test suite has the case as a hidden test. On a Pi 5 with the system Mesa:
git clone https://github.com/garana/locus && cd locus
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j4
# LOCUS_VK_SPLIT=0 turns off the application's workaround.
LOCUS_VK_SPLIT=0 LOCUS_VK_STRESS_ITERS=300 \
./build/tests/locus_tests "vulkan conv2d repeats its bits"
journalctl -k | grep "Resetting GPU"
The case runs five convolution dispatches of different sizes in turn, 300 rounds, on fixed inputs, and reports every round whose output differs from the first one's. One run gave 15 wrong rounds and 15 resets in the kernel log; with the workaround on (no LOCUS_VK_SPLIT), 0 and 0. A wrong round prints like:
conv 32 -> 32 k 3 at 149: round 33: 2243 of 691488 outputs differ, channels 0 .. 4, pixels 21248 .. 21608
Standalone attempts did not trigger it: a 338-workgroup dispatch of a pure ALU loop, and one with 12 KiB of shared memory and barriers in the loop, both running about 280 ms, gave 0 resets in 600 rounds each, with and without a short dispatch between rounds. The convolution kernel also streams its input and weights from memory (about 3 MB a dispatch), so memory traffic may be part of the condition.
Device (s)
Raspberry Pi 5
System
- Raspberry Pi 5 Model B Rev 1.0, 4 GB; V3D 7.1.7.0
- OS: Raspberry Pi reference 2024-03-15 (Debian 12, bookworm), aarch64
- Firmware: 2025/12/08 19:29:54, version 2226a853 (release) (embedded)
- Kernel: 6.12.75+rpt-rpi-2712, #1 SMP PREEMPT Debian 1:6.12.75-1+rpt1~bookworm (2026-03-11)
- Mesa 24.2.8-1~bpo12+rpt4 (V3DV), Vulkan 1.2.289
Logs
Once per dropped job:
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
v3d 1002000000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
Additional context
Workaround: recording the same grid as several vkCmdDispatchBase calls of at most 32 workgroups each (v3dv runs each as its own CSD job, so every job is short) avoids it completely: 0 resets and 0 wrong results in the same 300 rounds and over a full test suite, with no measurable change in total time.
Our reading of the cause (unconfirmed): as we read drivers/gpu/drm/v3d/v3d_sched.c, the CSD timeout handler re-arms the timer only while V3D_CSD_CURRENT_CFG4 (the batches remaining) changes between two checks. Near the end of a long job, with the last batches already dispatched and still running, that counter may stop changing for one timer period although the job is progressing, and the job is then taken for hung. If so, the last batches in flight are lost at the reset.
The measurements are in https://github.com/garana/locus/issues/682.
- 主要言語
- C
- スター
- 13.3k
- フォーク
- 5.5k
- 平均マージ
- 4日 10時間
- マージ済み PR(30日)
- 31
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
raspberrypi/linux のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
raspberrypi/linux#7415 · コメント 2 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
rp1-cfe doesn't forward V4L2_EVENT_SOURCE_CHANGE event from csi-2 sensor driver to userspace appオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
raspberrypi/linux#7399 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 82/100
raspberrypi/linux#7357 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
raspberrypi/linux#7054 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
raspberrypi/linux#7770 ·
メンテナーはふだん 1 日以内に返信
raspberrypi/linux の issue をすべて見る
似ている issue
-
難易度 1/5 1時間未満 初心者へのやさしさ 82/100
LedgerHQ/app-bitcoin-legacy#306 ·
-
[issue]: Ventoy2Disk.sh can leave the end of VTOYEFI unwritten (xzcat | dd without iflag=fullblock)オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 73/100
prusa3d/Prusa-Firmware-Buddy#5629 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 84/100
メンテナーはふだん 3 日以内に返信
-
Issue PRL on Windowsオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信