Intermittent CUDA context initialization OOM with NVIDIA sysmem mmap rejection at vm_insert_page zero-refcount guard

Start with the exact Ubuntu 6.8.0-142 kernel source and NVIDIA 610.57.04 module paths for vm_insert_page(), nvidia_mmap(), and the system-page pool; the issue provides build identities and a public evidence package. Review the recorded return chain and coverage limitations first. The requested outcome is maintainer guidance on the ownership contract, supported per-page observability, error translation, and relevant fixes; this report does not identify a code change or test to run.

NVIDIA Open GPU Kernel Modules Version

610.57.04

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04

Kernel Release

6.8.0-142-generic

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

Geforce RTX 5060 Ti 16GB

Describe the bug

Intermittent CUDA context initialization OOM with NVIDIA sysmem mmap rejection at vm_insert_page zero-refcount guard

Summary

A model-free CUDA context initialization has produced two NVIDIA system-memory mapping failures, followed by a CUDA out-of-memory exception. Complete target-thread function-boundary capture, matched to the exact Ubuntu kernel source and binary control flow, identifies the early page_count(page) == 0 rejection in vm_insert_page() for these calls.

We are requesting guidance on the system-page pool ownership contract and supported per-page lifetime instrumentation. We do not claim use-after-free, premature free, a defective pool implementation, NVIDIA-caused memory corruption, or a common cause with a separate historical page-cache issue.

Exact software identity

Component	Recorded identity
OS/kernel	Ubuntu; 6.8.0-142-generic, source package linux 6.8.0-142.142
Loaded kernel GNU build ID	da47fe975e5f116df7daae4314c20fcb0dd564e0
NVIDIA kernel modules	610.57.04, open
Loaded NVIDIA module GNU build ID	89639e51c5a84fa93e4fd79e050eb7f08701c026
PyTorch in recorded worker	2.11.0+cu130
Kernel taint	12288, unchanged during the recorded experiment

Installed driver source and matching module disassembly were audited. The prior saved module ELF matches the installed executable ELF; the installed file also has an appended module signature. Ubuntu packaged source and relevant exact-build branches were matched; upstream Linux 6.8 was not substituted for Ubuntu's patched build. No claim of a reproducible rebuild of the entire kernel is made.

Minimal triggering operation — not a deterministic reproducer

The diagnostic worker ran the equivalent of the following once, in the recorded existing Python environment:

import torch
torch.cuda.set_device(0)

There was no preliminary torch.cuda.is_available() call, model load, tensor stress allocation, JIT, CUDA graph capture, inference request or training. Tracing/monitoring was ready before a gate allowed this call. The call budget was one attempt and at most 30 seconds. The worker saved its terminal exception and exited itself with status 2; the monitor did not terminate it.

This is a minimal operation observed to trigger the failure, not a guaranteed reproducer on a freshly booted host. A prior context-only attempt in the same investigation sequence succeeded; its trace buffer overran, so it cannot prove absence of all helper failures. The detailed allocation experiment below was performed in an already-running boot with preexisting pool history unknown. No clean-boot trial was performed for the proposed escalation.

Observed return chain

Same-thread nested calls and return pairing establish two instances of:

vm_insert_page() returns -22 (-EINVAL)
  -> nvidia_mmap() returns -11 (-EAGAIN)
  -> mmap(/dev/nvidiactl, ...) returns EAGAIN

In both failed VM calls, validate_page_before_insert() was not entered. No missed probes or ring losses were reported in this capture. Source-derived interpretation: the exact kernel's pre-validation guard page_count(page) == 0 rejected the page. We did not sample the page's refcount field or read struct page contents. The count uses the exact kernel's folio/head resolution semantics.

The corresponding kernel message occurred twice:

NVRM: Userspace mapping creation failed [-22]!

Terminal userspace result:

torch.AcceleratorError: CUDA error: out of memory
operation: torch.cuda.set_device(0)

The mapping failures precede the terminal CUDA error during the same initialization. The kernel helper-to-syscall path is structurally captured. The proprietary userspace conversion from mapping failure to CUDA OOM is not directly traced. Physical VRAM exhaustion is not established by the error name.

Allocation registration and reuse evidence

Pointers are consistently replaced with opaque aliases in the public evidence. The same observed page token PAGE_P1 was rejected twice. Allocation token ALLOC_A is split into two observed registration generations; recurrence of a pointer is not treated as recurrence of the same lifetime.

Recorded quantity	Generation 1	Generation 2
nv_alloc_system_pages return	NV_OK / 0	NV_OK / 0
nv_alloc_set_page registrations	1516	1516
Registration index of PAGE_P1	632	763
Attempted mapping prefix matching registrations	633/633	764/764
Failed vm_insert_page return	-22	-22
Validation helper reached for failed call	No	No
Driver release path after failure	Observed	Observed

The registration functions supplied the same virtual/translated physical token in the two generations. Correlation to the opaque page token is source-derived using two independently nested, same-generation allocation/translation calibration chains and verified sparse-vmemmap arithmetic. No arbitrary page/PFN dereference or physical-memory read was performed. Function completion was captured; page-table stores were not read back.

Neither of these two system-allocation invocations contained a fresh __alloc_pages or __get_free_pages call in the fully captured target-thread nesting. Given the exact driver's source flow, this supports supply from the existing pool. It does not establish when that page originally entered the pool, whether pool contents were initially valid, or who changed its reference state.

After the first rejection, nv_free_pages -> nv_free_system_pages executes before the second generation registers the token again. Current-generation release happens after each failure. No matching target-thread __free_pages event for PAGE_P1 was captured. This does not exclude a prior or cross-thread free. A release request/system release is not automatically a physical free because memory can be returned to a pool.

Trace coverage and limits

Metric, detailed allocation experiment	Value
vm_insert_page entries/returns	14214 / 14214
vm_insert_page negative returns	2, both -22
validation entries/returns	14212 / 14212
validation negative returns	0
nvidia_mmap calls	27; two -11
syscall EAGAIN results	2
NVRM mapping error messages	2
Named full-symbol probes	21
Per-probe misses	0
Ring overruns/dropped/unread events	0
Entry/return nesting	Complete in fixed target TID

Capture used named tracefs kprobe/kretprobe events, individual kprobe_profile counters, per-CPU ring statistics, and paired/nested event analysis. Worker PID/TGID/starttime/cgroup identity was verified before the CUDA gate. No unsupported instruction-offset probes were used. Instrumentation can perturb timing and resource allocation.

This coverage is not complete cross-thread lifetime coverage. Pre-trace allocations, pool-worker activity and other-thread frees remain unknown. Existing full symbols can expose worker/shrinker activity and some release boundaries, but per-page pool admission and clean/dirty transfers are inlined. We did not infer unobserved ownership transitions from coarse function returns. A proposed clean-state experiment was withheld because the mandatory ownership-coverage gap remains.

Safety and independent issue

In the detailed experiment there was no new kernel Oops/Bad page/Bad RSS, Xid, kernel OOM or SIGSEGV. Application CUDA OOM is reported separately. Five existing file sentinels matched expected hashes through buffered, mmap and O_DIRECT reads before and after the attempt.

Sampled GPU-process memory never registered an allocation in the polling samples, but the short failed initialization can evade polling; this is not proof of zero instantaneous GPU memory usage. Resource monitoring remained within its sampled bounds.

A historical buffered/mmap-versus-O_DIRECT bit-mismatch issue remains separate and unresolved. These findings do not establish a common cause. No reboot, driver change, pool parameter change, cache repair, model load or repeated CUDA attempt was performed during the detailed allocation experiment.

Evidence labels

• OBSERVED: signed helper returns, paired syscall errors, missing validation entry with complete target-thread probe accounting, terminal CUDA exception, registration/release ordering and repeated opaque tokens.
• SOURCE_DERIVED: zero-refcount guard selection; physical/page-token correlation; preexisting pool supply under the audited exact source flow.
• INFERRED: a persistent invalid ownership/reference state is a candidate explanation, not a proven driver defect.
• UNKNOWN: original allocation/pool admission history, actual refcount transition, cross-thread release, error translation inside CUDA userspace, ultimate cause and relationship to historical page-cache corruption.

Requested guidance

1. Zero-refcount page: What circumstances can lead the driver to pass a page to vm_insert_page() whose exact-kernel page_count(page) == 0 guard rejects it? This guard outcome is source-derived from complete function-boundary capture, not a direct page-field sample.
2. Ownership contract: How are ownership and references maintained across the NVIDIA system-page pool, including allocation, clean/dirty lists, reuse, failed mmap cleanup, worker activity and shrinker reclaim?
3. Supported observability: Are supported tracepoints or debug interfaces available for per-page pool admission, reuse, transfer and release, with allocation generations and cross-thread attribution? Several pool helpers are inlined in this build.
4. Error translation: Can a failed vm_insert_page() followed by nvidia_mmap() returning -EAGAIN and syscall mmap reporting EAGAIN be translated into CUDA error out of memory during torch.cuda.set_device(0)? Which userspace/kernel diagnostic would establish that translation?
5. Known issues / fixes: Are there known issues or relevant fixes for NVIDIA 610.57.04 open modules with Ubuntu 6.8.0-142-generic (source package 6.8.0-142.142), matching the exact build IDs below? We have not independently established a matching known issue or fix.
6. Differential diagnostics: What additional diagnostics would distinguish a driver allocation/lifetime defect from kernel memory-management corruption or memory/hardware corruption without arbitrary page/PFN dereferences, unsupported offset probes or fault injection?

Attachments

PUBLIC_EVIDENCE.json, SYSCALL_EXCERPT.txt, COVERAGE_LIMITATIONS.md and the package manifest contain only selected, redacted evidence. Actual addresses, PIDs, boot ID, host/user identity, internal paths/network data, complete process environment, model/cache contents and private raw dumps are excluded. Full private originals are preserved locally and can be reviewed selectively if additional evidence is needed.

public_evidence_package.zip

To Reproduce

Environment

• Ubuntu Linux, kernel 6.8.0-142-generic
• NVIDIA Open GPU Kernel Modules 610.57.04
• PyTorch 2.11.0+cu130
• NVIDIA RTX 5060 Ti 16GB

Steps to Reproduce

1. Boot Ubuntu with NVIDIA Open GPU Kernel Modules loaded.
2. Activate the existing Python virtual environment containing PyTorch with CUDA support.
3. Execute the following minimal Python code:

import torch
torch.cuda.set_device(0)

4. Inspect the NVIDIA kernel messages using journalctl -k or dmesg.

Observed Behavior

During CUDA initialization, the NVIDIA driver intermittently reports:

NVRM: Userspace mapping creation failed [-22]!

Kernel tracing confirmed the following error propagation:

vm_insert_page() → -EINVAL (-22)
nvidia_mmap() → -EAGAIN (-11)
mmap(/dev/nvidiactl, ...) → EAGAIN

The CUDA initialization subsequently failed with:

torch.AcceleratorError: CUDA error: out of memory

Function-boundary tracing and matching Ubuntu kernel source indicate that the page_count(page) == 0 guard was responsible for the rejected mappings.

The same opaque page identifier was observed across two allocation-registration generations, but its original allocation and reference-count history could not be established.

Expected Behavior

CUDA context initialization should complete without unexpected system-memory mapping failures or CUDA out-of-memory errors when sufficient GPU memory is available.

Reproducibility

Intermittent, not deterministic. The operation has both succeeded and failed during controlled experiments. The failure has not been established as reproducible on every clean boot.

No model loading, JIT compilation, CUDA graph capture, inference, or training is required to observe the reported mapping failure.

Bug Incidence

Sometimes

nvidia-bug-report.log.gz

uname: Linux [REDACTED_HOSTNAME] 6.8.0-142-generic x86_64
NVIDIA-SMI version: 610.57.04
KMD version: 610.57.04
CUDA UMD version: 13.3
GPU: NVIDIA GeForce RTX 5060 Ti, 16311 MiB
OS: Ubuntu 24.04.5 LTS

Oct 08 10:22:46 [REDACTED_HOSTNAME] kernel: NVRM: Userspace mapping creation failed [-22]!
Oct 08 10:22:46 [REDACTED_HOSTNAME] kernel: NVRM: Userspace mapping creation failed [-22]!
Oct 08 11:09:39 [REDACTED_HOSTNAME] kernel: NVRM: Userspace mapping creation failed [-22]!
Oct 08 11:32:14 [REDACTED_HOSTNAME] kernel: NVRM: Userspace mapping creation failed [-22]!
Oct 08 11:32:14 [REDACTED_HOSTNAME] kernel: NVRM: Userspace mapping creation failed [-22]!

nvidia-bug-report.redacted.log.gz

More Info

No response