NV_ERR_NO_MEMORY (0x51) at first CUDA context on 580.178.04 (RTX PRO 6000 Blackwell): LEVEL_ERROR on a successful 4K fallback?

未关闭
  • 哪里不对？
#1,502 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
38/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
c, linux
领域
operating-systems

调研方向

Start by checking the tagged 580.178.04 source paths named in the report: uvm_rm_mem.c, kern_gmmu.c, mem_desc.c, and system_mem.c. Trace the 2 MiB allocation request and fallback to 4 KiB, then compare the logged failure with the successful context creation described here. Done means confirming whether the error-level log is expected and whether fallback failure reaches a visible context creation error; the report does not identify a test.

由索引模型根据 Issue 内容生成。

描述

Follow-up to #848 (closed with "open a new issue"). On 580.178.04 we see NV_ERR_NO_MEMORY at the first CUDA context of a process, apparently followed by a successful fallback. Our question is whether the LEVEL_ERROR print on a successful fallback can be demoted, and whether UVM could avoid asking for 2 MiB contiguous sysmem at GPU registration.

Environment

  • GPU: NVIDIA RTX PRO 6000 Blackwell (97887 MiB), one GPU per host, no MIG
  • Driver: 580.178.04, open kernel module
  • Kernel: 7.0.0-38-generic, x86_64
  • Persistence mode enabled (nvidia-persistenced active), memlock unlimited
  • THP: enabled=madvise, defrag=madvise
  • CUDA workloads run in containers with no cgroup memory limit

What we see

About 13-17 times per 24 h, at the first CUDA context creation of a container:

  • NVRM: Out of memory [NV_ERR_NO_MEMORY] (0x51) from _memdescAllocInternal (mem_desc.c)
  • the same status from sysmemConstruct (system_mem.c:356)
  • the same status from nvGpuOpsAllocPhysical
  • on some events also Allocation failed with big page size, retrying with default page size from _kgmmuClientShadowFaultBufferPagesAllocate

No Xid is logged and the GPU is not reset. In the cases we can join to the job outcome the job finishes normally with exit code 0.

Host memory is not exhausted at those moments (MemAvailable 79-90 GiB). It does look fragmented: /proc/vmstat shows compact_fail about 15.7 M vs compact_success about 1.7 M, and order-9/10 free blocks in /proc/buddyinfo drop to 0 at times. In our data the events cluster when a large memory-heavy process has just run (3 of 51 container starts during such runs, 0 of 59 outside them).

What we read in the source (tag 580.178.04)

This is our reading, please correct it if wrong.

  • uvm_rm_mem_alloc (uvm_rm_mem.c) asks for a 2M page size when size >= 2 MiB, else 64K, and retries with the default page size on NV_ERR_NO_MEMORY. That retry message goes through UVM_ERR_PRINT, so it is silent in release builds.
  • The request reaches nvGpuOpsAllocPhysical and then sysmemConstruct, which fails at LEVEL_ERROR for an explicit page size. This is the line that shows up in dmesg.
  • _kgmmuClientShadowFaultBufferPagesAllocate (kern_gmmu.c) tries RM_PAGE_SIZE_HUGE and retries at 4K, with a Bug 5042223 TODO.
  • uvm_rm_mem.c and kern_gmmu.c are identical between 580.178.04 and 580.196.07.

In #848 (2025-12-16) a collaborator wrote that a similar issue "was fixed in 580.xx ... where in cases of fragmented system memory, it would retry with smaller page sizes". If that retry is the intended fix, then the message looks like a normal fallback to us, but it is logged at error level, and health tooling that watches kernel logs cannot tell it apart from a real failure.

Questions

  1. Is the LEVEL_ERROR print on a successful fallback expected, and could it be demoted (or the fallback be made quiet) in a 580 branch release?
  2. Can UVM avoid requesting 2 MiB physically contiguous sysmem when a GPU is registered, since the 4 KiB path works?
  3. Is the Bug 5042223 TODO the tracker for this, and is there a driver version that changes the behaviour?
  4. If the 4 KiB fallback also fails, does context creation fail with a visible error? We have not been able to provoke that.

We can provide /proc/buddyinfo, /proc/vmstat and nvidia-bug-report.sh output if useful.

主要语言
C
星标
17.6k
派生
2k
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/open-gpu-kernel-modules 的其他 Issue

查看 NVIDIA/open-gpu-kernel-modules 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。