NV_ERR_NO_MEMORY (0x51) at first CUDA context on 580.178.04 (RTX PRO 6000 Blackwell): LEVEL_ERROR on a successful 4K fallback?

オープン
  • 何か問題がありますか？
#1,502 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
38/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
c, linux
領域
operating-systems

調査の方向性

Start by checking the tagged 580.178.04 source paths named in the report: uvm_rm_mem.c, kern_gmmu.c, mem_desc.c, and system_mem.c. Trace the 2 MiB allocation request and fallback to 4 KiB, then compare the logged failure with the successful context creation described here. Done means confirming whether the error-level log is expected and whether fallback failure reaches a visible context creation error; the report does not identify a test.

索引モデルが issue の本文から書いたものです。

説明

Follow-up to #848 (closed with "open a new issue"). On 580.178.04 we see NV_ERR_NO_MEMORY at the first CUDA context of a process, apparently followed by a successful fallback. Our question is whether the LEVEL_ERROR print on a successful fallback can be demoted, and whether UVM could avoid asking for 2 MiB contiguous sysmem at GPU registration.

Environment

  • GPU: NVIDIA RTX PRO 6000 Blackwell (97887 MiB), one GPU per host, no MIG
  • Driver: 580.178.04, open kernel module
  • Kernel: 7.0.0-38-generic, x86_64
  • Persistence mode enabled (nvidia-persistenced active), memlock unlimited
  • THP: enabled=madvise, defrag=madvise
  • CUDA workloads run in containers with no cgroup memory limit

What we see

About 13-17 times per 24 h, at the first CUDA context creation of a container:

  • NVRM: Out of memory [NV_ERR_NO_MEMORY] (0x51) from _memdescAllocInternal (mem_desc.c)
  • the same status from sysmemConstruct (system_mem.c:356)
  • the same status from nvGpuOpsAllocPhysical
  • on some events also Allocation failed with big page size, retrying with default page size from _kgmmuClientShadowFaultBufferPagesAllocate

No Xid is logged and the GPU is not reset. In the cases we can join to the job outcome the job finishes normally with exit code 0.

Host memory is not exhausted at those moments (MemAvailable 79-90 GiB). It does look fragmented: /proc/vmstat shows compact_fail about 15.7 M vs compact_success about 1.7 M, and order-9/10 free blocks in /proc/buddyinfo drop to 0 at times. In our data the events cluster when a large memory-heavy process has just run (3 of 51 container starts during such runs, 0 of 59 outside them).

What we read in the source (tag 580.178.04)

This is our reading, please correct it if wrong.

  • uvm_rm_mem_alloc (uvm_rm_mem.c) asks for a 2M page size when size >= 2 MiB, else 64K, and retries with the default page size on NV_ERR_NO_MEMORY. That retry message goes through UVM_ERR_PRINT, so it is silent in release builds.
  • The request reaches nvGpuOpsAllocPhysical and then sysmemConstruct, which fails at LEVEL_ERROR for an explicit page size. This is the line that shows up in dmesg.
  • _kgmmuClientShadowFaultBufferPagesAllocate (kern_gmmu.c) tries RM_PAGE_SIZE_HUGE and retries at 4K, with a Bug 5042223 TODO.
  • uvm_rm_mem.c and kern_gmmu.c are identical between 580.178.04 and 580.196.07.

In #848 (2025-12-16) a collaborator wrote that a similar issue "was fixed in 580.xx ... where in cases of fragmented system memory, it would retry with smaller page sizes". If that retry is the intended fix, then the message looks like a normal fallback to us, but it is logged at error level, and health tooling that watches kernel logs cannot tell it apart from a real failure.

Questions

  1. Is the LEVEL_ERROR print on a successful fallback expected, and could it be demoted (or the fallback be made quiet) in a 580 branch release?
  2. Can UVM avoid requesting 2 MiB physically contiguous sysmem when a GPU is registered, since the 4 KiB path works?
  3. Is the Bug 5042223 TODO the tracker for this, and is there a driver version that changes the behaviour?
  4. If the 4 KiB fallback also fails, does context creation fail with a visible error? We have not been able to provoke that.

We can provide /proc/buddyinfo, /proc/vmstat and nvidia-bug-report.sh output if useful.

主要言語
C
スター
17.5k
フォーク
1.9k
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/open-gpu-kernel-modules のほかの issue

NVIDIA/open-gpu-kernel-modules の issue をすべて見る

似ている issue

C の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。