NV_ERR_NO_MEMORY (0x51) at first CUDA context on 580.178.04 (RTX PRO 6000 Blackwell): LEVEL_ERROR on a successful 4K fallback?

Open
  • Is something wrong?
#1,502 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, linux
Domain
operating-systems

Research direction

Start by checking the tagged 580.178.04 source paths named in the report: uvm_rm_mem.c, kern_gmmu.c, mem_desc.c, and system_mem.c. Trace the 2 MiB allocation request and fallback to 4 KiB, then compare the logged failure with the successful context creation described here. Done means confirming whether the error-level log is expected and whether fallback failure reaches a visible context creation error; the report does not identify a test.

Written by the indexing model from the issue text.

Description

Follow-up to #848 (closed with "open a new issue"). On 580.178.04 we see NV_ERR_NO_MEMORY at the first CUDA context of a process, apparently followed by a successful fallback. Our question is whether the LEVEL_ERROR print on a successful fallback can be demoted, and whether UVM could avoid asking for 2 MiB contiguous sysmem at GPU registration.

Environment

  • GPU: NVIDIA RTX PRO 6000 Blackwell (97887 MiB), one GPU per host, no MIG
  • Driver: 580.178.04, open kernel module
  • Kernel: 7.0.0-38-generic, x86_64
  • Persistence mode enabled (nvidia-persistenced active), memlock unlimited
  • THP: enabled=madvise, defrag=madvise
  • CUDA workloads run in containers with no cgroup memory limit

What we see

About 13-17 times per 24 h, at the first CUDA context creation of a container:

  • NVRM: Out of memory [NV_ERR_NO_MEMORY] (0x51) from _memdescAllocInternal (mem_desc.c)
  • the same status from sysmemConstruct (system_mem.c:356)
  • the same status from nvGpuOpsAllocPhysical
  • on some events also Allocation failed with big page size, retrying with default page size from _kgmmuClientShadowFaultBufferPagesAllocate

No Xid is logged and the GPU is not reset. In the cases we can join to the job outcome the job finishes normally with exit code 0.

Host memory is not exhausted at those moments (MemAvailable 79-90 GiB). It does look fragmented: /proc/vmstat shows compact_fail about 15.7 M vs compact_success about 1.7 M, and order-9/10 free blocks in /proc/buddyinfo drop to 0 at times. In our data the events cluster when a large memory-heavy process has just run (3 of 51 container starts during such runs, 0 of 59 outside them).

What we read in the source (tag 580.178.04)

This is our reading, please correct it if wrong.

  • uvm_rm_mem_alloc (uvm_rm_mem.c) asks for a 2M page size when size >= 2 MiB, else 64K, and retries with the default page size on NV_ERR_NO_MEMORY. That retry message goes through UVM_ERR_PRINT, so it is silent in release builds.
  • The request reaches nvGpuOpsAllocPhysical and then sysmemConstruct, which fails at LEVEL_ERROR for an explicit page size. This is the line that shows up in dmesg.
  • _kgmmuClientShadowFaultBufferPagesAllocate (kern_gmmu.c) tries RM_PAGE_SIZE_HUGE and retries at 4K, with a Bug 5042223 TODO.
  • uvm_rm_mem.c and kern_gmmu.c are identical between 580.178.04 and 580.196.07.

In #848 (2025-12-16) a collaborator wrote that a similar issue "was fixed in 580.xx ... where in cases of fragmented system memory, it would retry with smaller page sizes". If that retry is the intended fix, then the message looks like a normal fallback to us, but it is logged at error level, and health tooling that watches kernel logs cannot tell it apart from a real failure.

Questions

  1. Is the LEVEL_ERROR print on a successful fallback expected, and could it be demoted (or the fallback be made quiet) in a 580 branch release?
  2. Can UVM avoid requesting 2 MiB physically contiguous sysmem when a GPU is registered, since the 4 KiB path works?
  3. Is the Bug 5042223 TODO the tracker for this, and is there a driver version that changes the behaviour?
  4. If the 4 KiB fallback also fails, does context creation fail with a visible error? We have not been able to provoke that.

We can provide /proc/buddyinfo, /proc/vmstat and nvidia-bug-report.sh output if useful.

Dominant language
C
Stars
17.5k
Forks
1.9k
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/open-gpu-kernel-modules

All issues in NVIDIA/open-gpu-kernel-modules

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.