After S3 resume, GSP RPCs (mostly PERF_BOOST) take seconds while the RM API lock is held → desktop-wide stalls, later Xid 119 → 62 → 154 (RTX 4070, 615.78.08 open)

Start by tracing the reported kgspRecvPoll_IMPL path through _issueRpcAndWait and rpcRmApiControl_GSP; the issue includes stack traces and Xid logs, while the privately submitted bug report may provide additional evidence. Compare behavior before and after S3 resume and inspect related reports #1271 and #1125. Done means identifying why GSP RPCs become slow and preventing the stalls and subsequent Xid failures.

NVIDIA Open GPU Kernel Modules Version

615.78.08 (nvidia-open-dkms 615.78.08-2, Arch/CachyOS)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

CachyOS (Arch-based), rolling

Kernel Release

7.2.9-2-cachyos (PREEMPT_DYNAMIC, clang 23.1.1)

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 4070 (AD104, 10de:2786), VBIOS 95.04.49.00.03

Describe the bug

After waking from S3 (deep), the GPU sometimes ends up in a state where RPCs to the GSP firmware take seconds to complete. The calling thread busy-polls in kgspRecvPoll while the RM API lock is held, so every other GPU client queues behind it. On a Wayland desktop that shows up as whole-desktop stutters and freezes (KWin Pageflip timed out, The main thread was hanging, hung_task reports for kwin_wayland and nvidia-modeset). In the worst case, the slow RPCs end in Xid 119 → Xid 62 → Xid 154 (PF FLR) and a black screen.

This looks like the same end state as #1271 (Ada, Arch, S3, Xid 119 GSP_RM_CONTROL → Xid 154), which is still open. What this report adds is the phase before the Xid: the bad state exists, and can be observed, for hours to days before anything is logged by the driver.

What the on-CPU lock holder is doing. I captured it with a small watcher that triggers SysRq-l when KWin blocks on the RM lock. This one is from 2026-10-20 14:20:31, on a boot that had been through 6 S3 cycles:

CPU: 3 UID: 1000 PID: 1338891 Comm: QSGRenderThread Tainted: G           OE       7.2.9-2-cachyos #1 PREEMPT(full)
RIP: 0010:osDevReadReg032+0x3a/0x70 [nvidia]
Call Trace:
 _regRead+0x113/0x140 [nvidia]
 intrIsVectorPending_TU102+0x15/0x30 [nvidia]
 intrGetPendingLowLatencyHwDisplayIntr_TU102+0x71/0xa0 [nvidia]
 kdispApplyAggressiveVblankHandlingWar_IMPL+0x3f/0x140 [nvidia]
 kgspRecvPoll_IMPL+0x1ee/0x8a0 [nvidia]
 _issueRpcAndWait+0x5a2/0x800 [nvidia]
 rpcRmApiControl_GSP+0x435/0x710 [nvidia]
 subdeviceCtrlCmdKPerfBoost_IMPL+0xdc/0x150 [nvidia]
 resControl_IMPL+0x171/0x1d0 [nvidia]
 gpuresControl_IMPL+0x53/0x90 [nvidia]
 serverControl+0x445/0x4a0 [nvidia]
 rmapiControlWithSecInfo+0x473/0x7a0 [nvidia]
 rmapiControlWithSecInfoTls+0x6f/0xc0 [nvidia]
 Nv04ControlWithSecInfo+0x9f/0xc0 [nvidia]
 RmIoctl+0x755/0xb60 [nvidia]
 rm_ioctl+0x248/0x490 [nvidia]
 nvidia_unlocked_ioctl+0x560/0x730 [nvidia]
 __x64_sys_ioctl+0x144/0x310
 do_syscall_64+0xa6/0x3e0

Other captures show the same poll reached through _kgspRpcDrainEvents → kgspHealthCheck_TU102 → crashcatEngineGetNextCrashReport_IMPL. While a stall is happening, nvidia-modeset/kthread_q sits in D state in nvkms_kthread_q_callback.

This is not specific to one application. Across 5 boots (Oct 18–22), SysRq-l caught about 90 on-CPU threads in kgspRecvPoll. They came from more than 20 different processes, mostly in NV2080_CTRL_CMD_PERF_BOOST:

PERF_BOOST dominates probably because nearly every GPU client issues it when it starts up or wakes up. The RPC that hangs is ordinary. The GSP is just slow to answer any RPC in this state.

Reproduction run

Kernel tainted G OE (only the NVIDIA module). Kernel 7.2.9-2-cachyos. The boot started 2026-10-18 23:10 from a full power drain (PSU switched off), so the GPU started out clean. That boot then ran until 10-22 with 12 S3 cycles and normal desktop use (browser, Steam/Proton games, Electron apps, a CUDA LLM server).

• Before the first suspend: 0 stalls.
• First wake 10-19 09:38 (after 9.0 h asleep) → first slow-RPC stall at 09:48:59. The holder was a Qt QSGRenderThread, on-CPU in kgspRecvPoll_IMPL ← _issueRpcAndWait ← rpcRmApiControl_GSP, with kwin_wayland and msedge waiting behind it.
• Over the whole boot the watcher triggered SysRq-l 77 times.
• End state after a 6.7 h sleep (wake 10-22 08:09):

08:44:17 INFO: task nvidia-modeset/:264 blocked for more than 51 seconds.   (reaches 71 s)
08:44:19 NVRM: Xid (PCI:0000:0a:00): 119, pid=4936, name=kwin_wayland, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 288825 (0x730280 0xc).
09:11:56 NVRM: Xid (PCI:0000:0a:00): 119, pid=4950, name=QSGRenderThread, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 293748 (0x20800a9a 0x8).
09:17:25 NVRM: Xid (PCI:0000:0a:00): 62, pid=898301, name=msedge, 3251c5ee 0000d2d8 00000000 202df764 202df468 202dc09e 202a064a 202db4b6
09:17:25 NVRM: GPU0 _kgspRpcGspEventPmuHalted: Received signal from GSP that PMU has halted.
09:17:25 NVRM: Xid (PCI:0000:0a:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
09:17:25 NVRM: Xid (PCI:0000:0a:00): 45, ... x18 (kwin_wayland, plasmashell, an Electron app)
10:11:36 nvidia-modeset: WARNING: GPU:0: Failed to determine which devices were hotplugged: 0x62   (repeats on every wake attempt; display stays black)

Stalls per wake on that boot (watcher triggers counted for 2 h after each wake, or until the next suspend):

wake (exit)  slept_h  window_min  stalls
10-19 09:38    9.0        26        1
10-19 11:13    1.1        16        0
10-19 13:09    1.7       120        0
10-19 18:56    0.2       120       11
10-20 08:17    9.1       120        2
10-20 10:53    0.1       120        0
10-20 19:35    0.5        42        2
10-20 20:39    0.4       120        0
10-21 08:20    8.6        77        0
10-21 09:40    0.1        17        2
10-21 11:59    2.0       120        1
10-22 08:09    6.7       120       46   -> Xid 119 x2, then PMU halted / Xid 62 / 154 / 45

It is not deterministic, and it doesn't follow sleep length cleanly: a 0.2 h sleep was followed by 11 stalls, while three sleeps of 8.6–9.1 h were followed by 0–2.

A warm reset does not clear the state. On this card a normal shutdown sometimes doesn't either. Removing standby power does.

• The boot right after the 10-22 hard reset had no suspend and logged 49 watcher lines in its first 7 minutes (up to 4 waiters: kwin_wayland, plasmashell, msedge, Xwayland, Discord).
• On 10-18 the card was already in the bad state when a boot started after an orderly poweroff. That boot never suspended and still reached the end state:

22:25:25 NVRM: Xid (PCI:0000:0a:00): 62, 3251c5ee 00005c10 00000000 202e3590 202e345e 202ff01a 202e6ef8 202e6c5c
22:25:25 NVRM: GPU0 _kgspRpcGspEventPmuHalted: Received signal from GSP that PMU has halted.
22:25:25 NVRM: Xid (PCI:0000:0a:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
         ... Xid 45 x23
22:57:09 nvidia-modeset: WARNING: GPU:0: Failed to determine which devices were hotplugged: 0x62
22:59:52 NVRM: Xid (PCI:0000:0a:00): 44, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_FECS faulted @ 0x1_20020000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE   (x4, during poweroff)

The next boot, after another orderly poweroff, stalled again within 2 minutes. While the GPU was in that state, nvidia-smi -q -d PERFORMANCE showed the clocks event reason Reliability: Active. Switching the PSU off for at least 60 s (so no standby power reaches the card) cleared it: on the next cold boot (the start of the reproduction run above) Reliability was Not Active, there were 0 stalls, and the card idled in P3 at 22 W. PCIe link was x16 Gen4 the whole time, with all AER counters at 0.

Configuration (all stock; nothing in /etc/modprobe.d)

NVreg_UseKernelSuspendNotifiers: 1   (615 default; nvidia-suspend/resume/hibernate services therefore disabled)
NVreg_PreserveVideoMemoryAllocations: 2
NVreg_DynamicPowerManagement: 3
NVreg_EnableGpuFirmware: 18
/sys/power/mem_sleep: s2idle [deep]

Platform: AMD Ryzen 5 5600X, ASUS ROG STRIX X570-I GAMING (BIOS 5044), KDE Plasma 6.8.0 / KWin 6.8.0 (Wayland). Display on DP-1.

The same post-resume degradation also happened on this machine in July 2026 on 610.43.02/610.43.03, but those logs have rotated out.

To Reproduce

1. Wayland desktop (KDE Plasma 6.8 here), RTX 4070, open 615.78.08, S3 deep.
2. Start from a cold boot with standby power removed, then use the machine normally with regular S3 suspends over a few days (12 cycles here).
3. After each resume, use the desktop normally: open a browser, start Steam, or start an app that creates a new GPU context.
4. Watch for KWin Pageflip timed out or The main thread was hanging. To see the holder, trigger SysRq-l while it stalls: the on-CPU thread sits in kgspRecvPoll_IMPL ← rpcRmApiControl_GSP for seconds.
5. If the session is left running, it can escalate to Xid 119 → 62 → 154. A warm reset does not clear it. Removing standby power does (PSU off for at least 60 s).

Bug Incidence

Sometimes

nvidia-bug-report.log.gz

Sent privately to [email protected], with this issue's link in the subject. It was captured 2026-10-18 23:04, on the boot after the 22:25 no-suspend PMU-halt episode above, and covers that episode's Xid 62/154/45/44 and the slow RPCs before it.

More Info

• Log excerpts are in the collapsed block below. They hold the Xid sequences, a full NMI backtrace, the list of on-CPU pollers, and the per-wake table.
• Related: #1271 (same end state, AD103, 610.43.03, S3), #1125 (GSP timeout on resume, several GPUs).
• Happy to test patches or debug builds, or to enable NVreg_EnableGpuFirmwareLogs or anything else useful.

# Excerpts: NVIDIA open 615.78.08, RTX 4070 (AD104, 10de:2786), S3 deep, kernel 7.2.9-2-cachyos, taint G OE. Times are +08:00.

## A. Reproduction run: boot 2026-10-18 23:10 (after PSU power drain) -> 2026-10-22 10:13, 12 S3 cycles
0 watcher stalls before the first suspend. 77 SysRq-l triggers in total.

End state, after the 6.7 h sleep (wake 10-22 08:09):
2026-10-22T01:26:36+08:00  PM: suspend entry (deep)
2026-10-22T08:09:41+08:00  PM: suspend exit
2026-10-22T08:44:17+08:00  INFO: task nvidia-modeset/:264 blocked for more than 51 seconds.
2026-10-22T08:44:19+08:00  NVRM: Xid (PCI:0000:0a:00): 119, pid=4936, name=kwin_wayland, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 288825 (0x730280 0xc).
2026-10-22T08:44:38+08:00  INFO: task nvidia-modeset/:264 blocked for more than 71 seconds.
2026-10-22T09:11:56+08:00  NVRM: Xid (PCI:0000:0a:00): 119, pid=4950, name=QSGRenderThread, Timeout after 45s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 293748 (0x20800a9a 0x8).
2026-10-22T09:17:25+08:00  NVRM: Xid (PCI:0000:0a:00): 62, pid=898301, name=msedge, 3251c5ee 0000d2d8 00000000 202df764 202df468 202dc09e 202a064a 202db4b6
2026-10-22T09:17:25+08:00  NVRM: GPU0 _kgspRpcGspEventPmuHalted: Received signal from GSP that PMU has halted.
2026-10-22T09:17:25+08:00  NVRM: Xid (PCI:0000:0a:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
(+ Xid 45 x18 at 09:17:25)
2026-10-22T10:11:36+08:00  nvidia-modeset: WARNING: GPU:0: Failed to determine which devices were hotplugged: 0x62
2026-10-22T10:11:37+08:00  nvidia-modeset: WARNING: GPU:0: Failed to determine which devices were hotplugged: 0x62
(0x62 hotplug warnings repeat on every wake attempt until the hard reset at 10:13)

Stalls per wake (2 h window, or until the next suspend):
wake (exit)  slept_h  window_min  stalls
10-19 09:38    9.0        26        1
10-19 11:13    1.1        16        0
10-19 13:09    1.7       120        0
10-19 18:56    0.2       120       11
10-20 08:17    9.1       120        2
10-20 10:53    0.1       120        0
10-20 19:35    0.5        42        2
10-20 20:39    0.4       120        0
10-21 08:20    8.6        77        0
10-21 09:40    0.1        17        2
10-21 11:59    2.0       120        1
10-22 08:09    6.7       120       46

## B. On-CPU GSP pollers caught by SysRq-l NMI backtrace, boots 10-18 21:57 -> 10-22 12:40 (count  comm: RPC path)
     22 msedge: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
     13 QSGRenderThread: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
     13 Discord: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      6 d3ddriverquery6: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      3 QSGRenderThread: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP
      3 ollama: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      3 llama-server: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      3 kwin_wayland: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- nvkms_call_rm
      3 CPMMListener: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      2 plasmashell: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP
      2 kscreenlocker_g: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 zenity: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 wolf: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 WaylandEventThr: kgspRecvPoll_IMPL <- rpcRmApiFree_GSP
      1 vulkandriverque: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 ThreadPoolSingl: kgspRecvPoll_IMPL <- rpcRmApiFree_GSP
      1 steamwebhelper: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 steam: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 QSGRenderThread: kgspRecvPoll_IMPL <- rpcRmApiAlloc_GSP
      1 kwin_wayland: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 ksmserver-logou: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 electron: kgspRecvPoll_IMPL <- rpcRmApiFree_GSP
      1 d3ddriverquery6: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP
      1 d3ddriverquery6: kgspRecvPoll_IMPL <- rpcRmApiAlloc_GSP
      1 codium: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL
      1 CJobMgr::m_Work: kgspRecvPoll_IMPL <- rpcRmApiControl_GSP <- subdeviceCtrlCmdKPerfBoost_IMPL

## C. Example on-CPU holder (2026-10-20 14:20:31, after 6 S3 cycles)
CPU: 3 UID: 1000 PID: 1338891 Comm: QSGRenderThread Tainted: G           OE       7.2.9-2-cachyos #1 PREEMPT(full)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: System manufacturer System Product Name/ROG STRIX X570-I GAMING, BIOS 5044 01/04/2026
RIP: 0010:osDevReadReg032+0x3a/0x70 [nvidia]
Call Trace:
 <TASK>
 _regRead+0x113/0x140 [nvidia]
 intrIsVectorPending_TU102+0x15/0x30 [nvidia]
 intrGetPendingLowLatencyHwDisplayIntr_TU102+0x71/0xa0 [nvidia]
 kdispApplyAggressiveVblankHandlingWar_IMPL+0x3f/0x140 [nvidia]
 kgspRecvPoll_IMPL+0x1ee/0x8a0 [nvidia]
 _issueRpcAndWait+0x5a2/0x800 [nvidia]
 rpcRmApiControl_GSP+0x435/0x710 [nvidia]
 subdeviceCtrlCmdKPerfBoost_IMPL+0xdc/0x150 [nvidia]
 resControl_IMPL+0x171/0x1d0 [nvidia]
 gpuresControl_IMPL+0x53/0x90 [nvidia]
 serverControl+0x445/0x4a0 [nvidia]
 rmapiControlWithSecInfo+0x473/0x7a0 [nvidia]
 rmapiControlWithSecInfoTls+0x6f/0xc0 [nvidia]
 Nv04ControlWithSecInfo+0x9f/0xc0 [nvidia]
 RmIoctl+0x755/0xb60 [nvidia]
 rm_ioctl+0x248/0x490 [nvidia]
 nvidia_unlocked_ioctl+0x560/0x730 [nvidia]
 __x64_sys_ioctl+0x144/0x310
 do_syscall_64+0xa6/0x3e0

## D. Boot after the 10-22 WARM (hard) reset, 0 suspends: 49 watcher lines in 7 min
2026-10-22T10:14:45+08:00 nv-lock-watch[5481]: [stall@10:14:44] waiters=1: ollama
2026-10-22T10:14:55+08:00 nv-lock-watch[10577]: [stall@10:14:54] waiters=1: kwin_wayland
2026-10-22T10:14:56+08:00 nv-lock-watch[12231]: [stall+1s@10:14:56] waiters=1: kwin_wayland
2026-10-22T10:15:05+08:00 nv-lock-watch[16734]: [stall@10:15:04] waiters=4: codium msedge kwin_wayland plasmashell
2026-10-22T10:15:07+08:00 nv-lock-watch[19872]: [stall+1s@10:15:06] waiters=4: Xwayland msedge Discord plasmashell
2026-10-22T10:15:17+08:00 nv-lock-watch[24608]: [stall@10:15:15] waiters=8: codium msedge Discord zenity gpu_encoder_hel gpu_encoder_hel gpu_encoder_hel kwin_wayland

## E. 2026-10-18 21:57 boot (started after an orderly poweroff, never suspended): already in the bad state, reaches PMU halt
2026-10-18T22:25:25+08:00  NVRM: Xid (PCI:0000:0a:00): 62, 3251c5ee 00005c10 00000000 202e3590 202e345e 202ff01a 202e6ef8 202e6c5c
2026-10-18T22:25:25+08:00  NVRM: GPU0 _kgspRpcGspEventPmuHalted: Received signal from GSP that PMU has halted.
2026-10-18T22:25:25+08:00  NVRM: Xid (PCI:0000:0a:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
2026-10-18T22:57:09+08:00  nvidia-modeset: WARNING: GPU:0: Failed to determine which devices were hotplugged: 0x62
2026-10-18T22:57:10+08:00  nvidia-modeset: WARNING: GPU:0: Failed to determine which devices were hotplugged: 0x62
2026-10-18T22:59:52+08:00  NVRM: Xid (PCI:0000:0a:00): 44, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_FECS faulted @ 0x1_20020000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
2026-10-18T22:59:53+08:00  NVRM: Xid (PCI:0000:0a:00): 44, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_FECS faulted @ 0x1_20020000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
2026-10-18T22:59:54+08:00  NVRM: Xid (PCI:0000:0a:00): 44, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_FECS faulted @ 0x1_20020000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
2026-10-18T22:59:54+08:00  NVRM: Xid (PCI:0000:0a:00): 44, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_FECS faulted @ 0x1_21e90000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
(+ Xid 45 x23 at 22:25:25; the Xid 44s happen during the orderly poweroff at 22:59)
Next boot (23:00, after another orderly poweroff): stalls again within 2 min; nvidia-smi clocks event reason Reliability: Active.
Then PSU switched off >= 60 s -> 23:10 cold boot: Reliability Not Active, 0 stalls, P3 idle 22 W (= start of section A).