Hot-unplug of a dGPU in a detachable base (Surface Book 3) deadlocks in nv_pci_remove_helper(); request opt-in to treat surprise-hotplug slot GPUs as external

Start with nv_pci_remove_helper() in kernel-open/nvidia/nv-pci.c (around L2475-2512), where the non-zero usage_count wait loop is, and with RmCheckForExternalGpu() in src/nvidia/arch/nvalloc/unix/src/osinit.c (around L426). The reproduction is the python hold-open snippet followed by a write to the GPU's sysfs remove node. Done means removal completes without waiting for clients, and the proposed RMForceExternalGpu key is a maintainer decision, not yet agreed.

NVIDIA Open GPU Kernel Modules Version

610.57.04 (also present in 615 source; the wait is in kernel-open/nvidia/nv-pci.c, the decision in the open RM src/nvidia/arch/nvalloc/unix/src/osinit.c)

Operating System and Version

Omarchy 4.0.4 (Arch Linux)

Kernel Release

7.2.5 (Arch-based linux-omarchy; mainline surface_aggregator / surface_hotplug drivers, no out-of-tree Surface patches)

Hardware

Microsoft Surface Book 3 15". The GeForce GTX 1660 Ti Max-Q (TU116M, 10de:2191) is inside the detachable keyboard base, behind Intel Ice Lake-LP PCIe root port #13 (8086:34b4, 0000:00:1d.4). The slot is hot-plug and surprise-removal capable (pciehp: Slot #12 ... HotPlug+ Surprise+). Hybrid laptop, Intel iGPU drives all displays, NVreg_DynamicPowerManagement=0x02, nvidia_drm modeset=1, runtime PM enabled by udev. The problem also reproduces with no NVIDIA-specific configuration.

Describe the bug

Detaching the base (or the base dropping out electrically) while any process has the GPU open hangs the kernel's hot-unplug path permanently:

1. nv_pci_remove_helper() (nv-pci.c ~L2475–2512 in 610.57.04) finds usage_count != 0, prints Attempting to remove device ... with non-zero usage count! and loops in os_delay(500) until the count reaches zero.
2. It runs inside the ACPI hotplug work (acpiphp_hotplug_notify → disable_slot → pci_stop_and_remove_bus_device → pci_device_remove) while holding pci_rescan_remove_lock and the device lock. The pciehp IRQ thread for the same slot then blocks in pciehp_unconfigure_device.
3. Because the device lock is never released, every later suspend blocks in dpm_prepare() and shutdown blocks in device_shutdown(). The machine has to be force-powered off.

This happened five times here (2026-09-27 ×2, 10-07, 10-10, 10-14). On 10-07 the GPU had already failed to come back from D3cold at suspend entry (Xid 79); the stuck removal then kept the system half-suspended, and the dead GPU stayed at full power and discharged the base battery from 86 % to 16 % in 37 minutes (~57 W) before the base cut out.

Additional observation: on 10-14 the wait continued for hours after no userspace process held any NVIDIA device node or mapping (checked /proc/*/fd and /proc/*/maps as root). The GPU had reported Xid 79 before the removal, so a usage_count reference appears to be leaked on the GPU-lost path.

Log excerpt (10-14):

pcieport 0000:00:1d.4: pciehp: Slot(12): Card not present
NVRM: Xid (PCI:0000:02:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:02:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
NVRM: Attempting to remove device 0000:02:00.0 with non-zero usage count!
INFO: task irq/125-pciehp:95 blocked for more than 122 seconds.
 pciehp_unconfigure_device+0x59/0x1b0
 pciehp_disable_slot+0x66/0x110
INFO: task irq/125-pciehp:95 is blocked on a mutex likely owned by task kworker/u32:0:1159805.
 nv_pci_remove_helper+0x3d0/0x5a0 [nvidia]
 acpiphp_hotplug_notify+0x152/0x2d0

The driver already has the right behaviour for this case: the comment at nv-pci.c ~L2470 says that for an eGPU "fall off the bus along with clients active is a valid scenario", and the wait is skipped when nv->is_external_gpu is set. RmCheckForExternalGpu() (osinit.c ~L426) sets that only when an upstream bridge is a GSP-approved Thunderbolt 3 bridge and a slot in the path is surprise-hot-plug capable. The Surface Book's root port satisfies the slot condition but is not a TB3 bridge, so a GPU that is physically designed to be unplugged is handled as a soldered one.

To Reproduce

No detach needed:

python3 -c "import os,time; os.open('/dev/nvidia0', os.O_RDWR); time.sleep(600)" &
echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove     # never returns on stock; logs the non-zero usage count message

Request

An opt-in to mark a GPU as hot-unpluggable when it is not behind a TB3 bridge, or extending RmCheckForExternalGpu() to accept a surprise-capable hot-plug slot on its own (or known detachable-base platforms such as Surface Book 2/3). The patch below adds an RM registry key, RMForceExternalGpu, checked at the top of RmCheckForExternalGpu(), plus one kernel log line. With options nvidia NVreg_RegistryDwords="RMForceExternalGpu=1" on this machine:

• removal with a process holding /dev/nvidia0, /dev/nvidiactl and the GPU's render node open completes in ~1 s (3/3 cycles), the root port goes to D3cold, a PCI rescan re-enumerates the GPU, the driver rebinds, Vulkan on the GPU works, and it returns to D3cold;
• nvidia-smi -q -d POWER,CLOCK,PERFORMANCE is identical to stock mode;
• real detach/attach via surface-dtx-daemon works with a handler that removes the GPU from the bus before unlatching.

Note: NVreg_RegistryDwordsPerDevice="pci=0000:02:00.0;RMForceExternalGpu=1" did not work, because per-device keys are parsed before the probing GPU is added to the device list; filed separately: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1450

Patch tested (applies to 610.57.04):

diff -ur a/kernel-open/nvidia/nv.c b/kernel-open/nvidia/nv.c
--- a/kernel-open/nvidia/nv.c	2026-10-14 19:32:14.397271845 -0600
+++ b/kernel-open/nvidia/nv.c	2026-10-14 19:32:14.417267491 -0600
@@ -1525,6 +1525,12 @@
         goto failed_release_irq;
     }
 
+    if (nv->is_external_gpu)
+    {
+        NV_DEV_PRINTF(NV_DBG_ERRORS, nv,
+                      "GPU is treated as external (hot-unpluggable): removal will not wait for open clients\n");
+    }
+
     /* Generate and cache the UUID for future callers */
     (void)rm_get_gpu_uuid_raw(sp, nv);
 
diff -ur a/src/nvidia/arch/nvalloc/unix/src/osinit.c b/src/nvidia/arch/nvalloc/unix/src/osinit.c
--- a/src/nvidia/arch/nvalloc/unix/src/osinit.c	2026-10-14 19:32:14.387642890 -0600
+++ b/src/nvidia/arch/nvalloc/unix/src/osinit.c	2026-10-14 19:32:14.416934508 -0600
@@ -440,6 +440,21 @@
     NV_STATUS status, rmStatus;
     NvBool bTb3Bridge = NV_FALSE, bSlotHotPlugSupport = NV_FALSE;
     NvBool iseGPUBridge = NV_FALSE;
+    NvU32 forceExternal = NV_REG_STR_RM_FORCE_EXTERNAL_GPU_DEFAULT;
+
+    //
+    // Allow the platform configuration to declare the GPU hot-unpluggable
+    // (e.g. a dGPU in a detachable keyboard base behind a surprise-capable
+    // PCIe root port), so that removal does not wait on open clients.
+    //
+    if ((osReadRegistryDword(pGpu, NV_REG_STR_RM_FORCE_EXTERNAL_GPU,
+                             &forceExternal) == NV_OK) &&
+        (forceExternal == NV_REG_STR_RM_FORCE_EXTERNAL_GPU_ENABLE))
+    {
+        NV_PRINTF(LEVEL_NOTICE,
+                  "RMForceExternalGpu set: treating GPU as external (hot-unpluggable)\n");
+        return NV_TRUE;
+    }
 
     pRmApi  = GPU_GET_PHYSICAL_RMAPI(pGpu);
     domain  = gpuGetDomain(pGpu);
diff -ur a/src/nvidia/interface/nvrm_registry.h b/src/nvidia/interface/nvrm_registry.h
--- a/src/nvidia/interface/nvrm_registry.h	2026-10-14 19:32:14.393229241 -0600
+++ b/src/nvidia/interface/nvrm_registry.h	2026-10-14 19:32:14.416705899 -0600
@@ -3433,4 +3433,21 @@
 #define NV_REG_STR_RM_USE_RW_API_LOCK_GET_MEM_ALIGNMENT_BUG_5785851_WAR_TRUE  (0x00000001)
 #define NV_REG_STR_RM_USE_RW_API_LOCK_GET_MEM_ALIGNMENT_BUG_5785851_WAR_DEFAULT NV_REG_STR_RM_USE_RW_API_LOCK_GET_MEM_ALIGNMENT_BUG_5785851_WAR_FALSE
 
+//
+// Type DWORD
+// Treat the GPU as an external (hot-unpluggable) GPU even when it is not behind
+// an approved Thunderbolt 3 bridge, e.g. the NVIDIA dGPU in the detachable base
+// of a Microsoft Surface Book 2/3, which sits behind a plain PCIe root port with
+// Hot-Plug Surprise capability. Without this, nv_pci_remove_helper() waits
+// forever for all clients to close when the base is detached, deadlocking PCI
+// hotplug (pciehp/acpiphp) and any later suspend or shutdown.
+//
+// 0 - Use the Thunderbolt 3 bridge detection only (default)
+// 1 - Treat the GPU as external / hot-unpluggable
+//
+#define NV_REG_STR_RM_FORCE_EXTERNAL_GPU                "RMForceExternalGpu"
+#define NV_REG_STR_RM_FORCE_EXTERNAL_GPU_DISABLE        (0x00000000)
+#define NV_REG_STR_RM_FORCE_EXTERNAL_GPU_ENABLE         (0x00000001)
+#define NV_REG_STR_RM_FORCE_EXTERNAL_GPU_DEFAULT        NV_REG_STR_RM_FORCE_EXTERNAL_GPU_DISABLE
+
 #endif // NVRM_REGISTRY_H

Bug Incidence

Always (any removal while a client holds the device).

nvidia-bug-report.log.gz

Available on request.