NULL pointer dereference in HMM after memory pressure causes gpu->cpu eviction

Start with the reported stack trace through uvm_hmm_must_use_sysmem, uvm_hmm_block_add_eviction_mappings, and block_add_eviction_mappings; the issue includes a CUDA reproducer and dmesg output. Reproduce HMM eviction under memory pressure, then trace how the deferred mapping path handles the VMA. Done means the reproducer no longer triggers a kernel NULL pointer dereference.

NVIDIA Open GPU Kernel Modules Version

615.78.08

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.4 LTS

Kernel Release

Linux REDACTED 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA RTX A4000 (UUID: GPU-REDACTED)

Describe the bug

This bug was found while we were testing the behavior of HMM memory under memory pressure. In some situations, after an HMM page is evicted due to memory pressure, the kernel will dereference a null pointer and needs to be reloaded or whole machine rebooted before it can be used again (kernel "oops"). The kernel thread is killed holding a lock and the observable symptom is that basically all future gpu operations hang (even nvidia-smi) until system reboot (it is not possible to unload/reload the kernel module because there are some held references to it that can't be released cause of the killed threads)

I won't claim to personally fully understand the chain of events and code that leads to the crash, but it is reproducible. Here's a description from our LLM:

Step-by-Step: How the Bug Happens

System memory moves into GPU VRAM: A program allocates normal Linux memory (mmap), sets the SetAccessedBy hint on it for the GPU, and touches it from the GPU so the driver migrates those pages into GPU VRAM.

VRAM fills up and triggers eviction: Later, the GPU runs low on VRAM (in the repro, by allocating 40 GiB of other memory). The driver decides to evict the older mmap pages from GPU VRAM back to system RAM.

Re-mapping is deferred to a background worker: Because the driver is honor-bound by the SetAccessedBy hint to keep those evicted pages mapped for the GPU, it needs to create new page-table mappings pointing to system RAM. However, it cannot safely lock the process's memory tables in the middle of an eviction. Instead, it hands the job off to a background kernel thread (block_add_eviction_mappings on the UVM global work queue).

The bug—the background worker skips the VMA lookup:
    When SetAccessedBy is called normally by a program, the driver looks up the Linux VMA (hmm.vma) and stores it in a context struct before creating the mappings.
    When the background eviction worker runs, it creates a blank context struct (hmm.vma = NULL) and calls the exact same mapping code without ever looking up the VMA.

The crash: Deep inside the mapping code, a function (uvm_hmm_must_use_sysmem) checks whether the Linux VMA has copy-on-write sharing (vma->anon_vma).
    In debug builds of the driver, there is an assertion (UVM_ASSERT(vma)) that catches the missing pointer.
    In release builds (what runs in production), that assertion is compiled out. The driver dereferences NULL + 0x48, causing a kernel NULL pointer dereference (Oops).

To Reproduce

nvcc to compile the following file then run:

// Reproducer for nvidia_uvm NULL pointer dereference in uvm_hmm_must_use_sysmem
// during deferred eviction mapping replay (block_add_eviction_mappings).
//
// Build:
//   nvcc -O2 -std=c++17 -o uvm_hmm_evict_oops uvm_hmm_evict_oops.cu
// Run:
//   ./uvm_hmm_evict_oops [device_id]

#include <cuda_runtime.h>
#include <cstdio>
#include <cstdlib>
#include <sys/mman.h>
#include <vector>

#define CHECK_CUDA(call)                                                       \
    do {                                                                       \
        cudaError_t err = (call);                                              \
        if (err != cudaSuccess) {                                              \
            std::fprintf(stderr, "%s:%d: %s failed: %s\n", __FILE__, __LINE__, \
                         #call, cudaGetErrorString(err));                      \
            std::exit(EXIT_FAILURE);                                           \
        }                                                                      \
    } while (0)

int main(int argc, char** argv) {
    int device = (argc > 1) ? std::atoi(argv[1]) : 0;
    CHECK_CUDA(cudaSetDevice(device));

    size_t free_bytes = 0, total_bytes = 0;
    CHECK_CUDA(cudaMemGetInfo(&free_bytes, &total_bytes));
    std::printf("GPU %d: %.2f GiB free / %.2f GiB total VRAM\n", device,
                free_bytes / double(1ULL << 30),
                total_bytes / double(1ULL << 30));

    // 1. Create a small (64 MiB) unregistered anonymous mmap (HMM backed).
    const size_t hmm_bytes = 64ULL << 20;
    void* hmm_ptr = mmap(nullptr, hmm_bytes, PROT_READ | PROT_WRITE,
                         MAP_ANONYMOUS | MAP_PRIVATE | MAP_NORESERVE, -1, 0);
    if (hmm_ptr == MAP_FAILED) {
        std::perror("mmap");
        return EXIT_FAILURE;
    }

    // Support both CUDA 13+ (cudaMemLocation struct) and CUDA <= 12 (int device)
#if CUDART_VERSION >= 13000
    cudaMemLocation gpu_loc = {};
    gpu_loc.type = cudaMemLocationTypeDevice;
    gpu_loc.id = device;
#else
    int gpu_loc = device;
#endif

    // 2. Set PreferredLocation(GPU) + AccessedBy(GPU), then fault it into VRAM.
    CHECK_CUDA(cudaMemAdvise(hmm_ptr, hmm_bytes, cudaMemAdviseSetPreferredLocation, gpu_loc));
    CHECK_CUDA(cudaMemAdvise(hmm_ptr, hmm_bytes, cudaMemAdviseSetAccessedBy, gpu_loc));
    CHECK_CUDA(cudaMemsetAsync(hmm_ptr, 0x11, hmm_bytes));
    CHECK_CUDA(cudaStreamSynchronize(nullptr));
    std::printf("Faulted 64 MiB HMM region into VRAM (%p)\n", hmm_ptr);

    // 3. Allocate and fault managed filler in 256 MiB chunks to exceed physical
    //    VRAM and force UVM to evict the idle HMM blocks.
    const size_t chunk_bytes = 256ULL << 20;
    const size_t target_fill = free_bytes + (2ULL << 30); // Exceed free VRAM by 2 GiB
    std::vector<void*> filler;

    std::printf("Filling VRAM to evict the HMM region...\n");
    for (size_t allocated = 0; allocated < target_fill; allocated += chunk_bytes) {
        void* p = nullptr;
        CHECK_CUDA(cudaMallocManaged(&p, chunk_bytes, cudaMemAttachGlobal));
        CHECK_CUDA(cudaMemsetAsync(p, 0, chunk_bytes));
        CHECK_CUDA(cudaStreamSynchronize(nullptr));
        filler.push_back(p);
    }

    std::printf("Done (if driver did not Oops, HMM was not enabled).\n");
    return 0;
}

ON my system, this causes the following dmesg log:

[4753223.700699] BUG: kernel NULL pointer dereference, address: 0000000000000070
[4753223.701791] #PF: supervisor read access in kernel mode
[4753223.702557] #PF: error_code(0x0000) - not-present page
[4753223.703317] PGD 0 P4D 0
[4753223.703716] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
[4753223.704453] CPU: 2 UID: 0 PID: 855 Comm: UVM global queu Tainted: G           OE      6.14.0-37-generic #37~24.04.1-Ubuntu
[4753223.706163] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[4753223.706939] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[4753223.708269] RIP: 0010:uvm_hmm_must_use_sysmem+0x13/0x80 [nvidia_uvm]
[4753223.709332] Code: 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 83 bf 30 0b 00 00 00 48 89 e5 74 56 <48> 83 7e 70 00 b8 01 00 00 00 74 0e 5d 31 d2 31 c9 31 f6 31 ff e9
[4753223.712002] RSP: 0018:ffffc90000e23b08 EFLAGS: 00010282
[4753223.712833] RAX: ffff888337d10000 RBX: ffffc9000308f008 RCX: 0000000000000000
[4753223.713951] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8882a3fb7350
[4753223.714989] RBP: ffffc90000e23b08 R08: 0000000000000000 R09: 0000000000000000
[4753223.716018] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8882a3d30c08
[4753223.717193] R13: ffff8882a3fb7350 R14: 0000000000000000 R15: ffff88810a098900
[4753223.718221] FS:  0000000000000000(0000) GS:ffff88856ad00000(0000) knlGS:0000000000000000
[4753223.719405] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4753223.720250] CR2: 0000000000000070 CR3: 0000000243e59000 CR4: 0000000000350ef0
[4753223.721287] Call Trace:
[4753223.721672]  <TASK>
[4753223.722058]  uvm_va_block_map+0x69e/0xf00 [nvidia_uvm]
[4753223.722959]  ? srso_return_thunk+0x5/0x5f
[4753223.723576]  ? srso_return_thunk+0x5/0x5f
[4753223.724723]  ? block_region_authorized_processors+0x42/0x160 [nvidia_uvm]
[4753223.726356]  ? uvm_va_block_page_resident_processors+0xb2/0x120 [nvidia_uvm]
[4753223.727854]  uvm_va_block_add_mappings+0x1ee/0x460 [nvidia_uvm]
[4753223.729259]  uvm_va_block_set_accessed_by_locked+0x22/0x60 [nvidia_uvm]
[4753223.730795]  hmm_set_accessed_by_start_end_locked+0x9e/0xe0 [nvidia_uvm]
[4753223.732229]  uvm_hmm_block_add_eviction_mappings+0x132/0x3c0 [nvidia_uvm]
[4753223.733839]  block_add_eviction_mappings+0xe5/0x490 [nvidia_uvm]
[4753223.735188]  ? srso_return_thunk+0x5/0x5f
[4753223.736214]  ? block_add_eviction_mappings+0xe5/0x490 [nvidia_uvm]
[4753223.737569]  block_add_eviction_mappings_entry+0x91/0xb0 [nvidia_uvm]
[4753223.739094]  _main_loop+0x81/0x140 [nvidia_uvm]
[4753223.740244]  ? __pfx__main_loop+0x10/0x10 [nvidia_uvm]
[4753223.741445]  kthread+0xfe/0x230
[4753223.742421]  ? __pfx_kthread+0x10/0x10
[4753223.743448]  ret_from_fork+0x47/0x70
[4753223.744373]  ? __pfx_kthread+0x10/0x10
[4753223.745326]  ret_from_fork_asm+0x1a/0x30
[4753223.746465]  </TASK>
[4753223.747195] Modules linked in: nbd nf_conntrack_netlink xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE bridge stp llc xfrm_user xfrm_algo xt_set ip_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat nf_tables vxlan ip6_udp_tunnel udp_tunnel overlay tls nvidia_uvm(OE) qrtr cfg80211 nvidia_drm(OE) nvidia_modeset(OE) nvidia(OE) intel_rapl_msr binfmt_misc intel_rapl_common drm_ttm_helper ttm nls_iso8859_1 i2c_piix4 i2c_smbus video wmi input_leds joydev mac_hid serio_raw sch_fq_codel dm_multipath msr efi_pstore nfnetlink dmi_sysfs qemu_fw_cfg ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 linear hid_generic polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 vga16fb sha1_ssse3 vgastate psmouse virtio_rng virtio_gpu virtio_dma_buf pata_acpi floppy usbhid hid aesni_intel crypto_simd cryptd
[4753223.761720] CR2: 0000000000000070
[4753223.762750] ---[ end trace 0000000000000000 ]---
[4753223.763974] RIP: 0010:uvm_hmm_must_use_sysmem+0x13/0x80 [nvidia_uvm]
[4753223.765422] Code: 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 83 bf 30 0b 00 00 00 48 89 e5 74 56 <48> 83 7e 70 00 b8 01 00 00 00 74 0e 5d 31 d2 31 c9 31 f6 31 ff e9
[4753223.769102] RSP: 0018:ffffc90000e23b08 EFLAGS: 00010282
[4753223.770335] RAX: ffff888337d10000 RBX: ffffc9000308f008 RCX: 0000000000000000
[4753223.771822] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8882a3fb7350
[4753223.773459] RBP: ffffc90000e23b08 R08: 0000000000000000 R09: 0000000000000000
[4753223.774964] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8882a3d30c08
[4753223.776586] R13: ffff8882a3fb7350 R14: 0000000000000000 R15: ffff88810a098900
[4753223.778065] FS:  0000000000000000(0000) GS:ffff88856ad00000(0000) knlGS:0000000000000000
[4753223.779780] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4753223.781087] CR2: 0000000000000070 CR3: 0000000243e59000 CR4: 0000000000350ef0
[4753223.782547] note: UVM global queu[855] exited with irqs disabled

Bug Incidence

Always

nvidia-bug-report.log.gz

not possible: nvidia-bug-report.sh hangs indefinitely after the bug is triggered because the kernel driver is in an inoperable state.

More Info

No response