NV_ERR_NO_MEMORY (0x51) at first CUDA context on 580.178.04 (RTX PRO 6000 Blackwell): LEVEL_ERROR on a successful 4K fallback?

Aperta
  • Qualcosa non va?
#1,502 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, linux
Ambito
operating-systems

Direzione di ricerca

Start by checking the tagged 580.178.04 source paths named in the report: uvm_rm_mem.c, kern_gmmu.c, mem_desc.c, and system_mem.c. Trace the 2 MiB allocation request and fallback to 4 KiB, then compare the logged failure with the successful context creation described here. Done means confirming whether the error-level log is expected and whether fallback failure reaches a visible context creation error; the report does not identify a test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Follow-up to #848 (closed with "open a new issue"). On 580.178.04 we see NV_ERR_NO_MEMORY at the first CUDA context of a process, apparently followed by a successful fallback. Our question is whether the LEVEL_ERROR print on a successful fallback can be demoted, and whether UVM could avoid asking for 2 MiB contiguous sysmem at GPU registration.

Environment

  • GPU: NVIDIA RTX PRO 6000 Blackwell (97887 MiB), one GPU per host, no MIG
  • Driver: 580.178.04, open kernel module
  • Kernel: 7.0.0-38-generic, x86_64
  • Persistence mode enabled (nvidia-persistenced active), memlock unlimited
  • THP: enabled=madvise, defrag=madvise
  • CUDA workloads run in containers with no cgroup memory limit

What we see

About 13-17 times per 24 h, at the first CUDA context creation of a container:

  • NVRM: Out of memory [NV_ERR_NO_MEMORY] (0x51) from _memdescAllocInternal (mem_desc.c)
  • the same status from sysmemConstruct (system_mem.c:356)
  • the same status from nvGpuOpsAllocPhysical
  • on some events also Allocation failed with big page size, retrying with default page size from _kgmmuClientShadowFaultBufferPagesAllocate

No Xid is logged and the GPU is not reset. In the cases we can join to the job outcome the job finishes normally with exit code 0.

Host memory is not exhausted at those moments (MemAvailable 79-90 GiB). It does look fragmented: /proc/vmstat shows compact_fail about 15.7 M vs compact_success about 1.7 M, and order-9/10 free blocks in /proc/buddyinfo drop to 0 at times. In our data the events cluster when a large memory-heavy process has just run (3 of 51 container starts during such runs, 0 of 59 outside them).

What we read in the source (tag 580.178.04)

This is our reading, please correct it if wrong.

  • uvm_rm_mem_alloc (uvm_rm_mem.c) asks for a 2M page size when size >= 2 MiB, else 64K, and retries with the default page size on NV_ERR_NO_MEMORY. That retry message goes through UVM_ERR_PRINT, so it is silent in release builds.
  • The request reaches nvGpuOpsAllocPhysical and then sysmemConstruct, which fails at LEVEL_ERROR for an explicit page size. This is the line that shows up in dmesg.
  • _kgmmuClientShadowFaultBufferPagesAllocate (kern_gmmu.c) tries RM_PAGE_SIZE_HUGE and retries at 4K, with a Bug 5042223 TODO.
  • uvm_rm_mem.c and kern_gmmu.c are identical between 580.178.04 and 580.196.07.

In #848 (2025-12-16) a collaborator wrote that a similar issue "was fixed in 580.xx ... where in cases of fragmented system memory, it would retry with smaller page sizes". If that retry is the intended fix, then the message looks like a normal fallback to us, but it is logged at error level, and health tooling that watches kernel logs cannot tell it apart from a real failure.

Questions

  1. Is the LEVEL_ERROR print on a successful fallback expected, and could it be demoted (or the fallback be made quiet) in a 580 branch release?
  2. Can UVM avoid requesting 2 MiB physically contiguous sysmem when a GPU is registered, since the 4 KiB path works?
  3. Is the Bug 5042223 TODO the tracker for this, and is there a driver version that changes the behaviour?
  4. If the 4 KiB fallback also fails, does context creation fail with a visible error? We have not been able to provoke that.

We can provide /proc/buddyinfo, /proc/vmstat and nvidia-bug-report.sh output if useful.

Lingua principale
C
Stelle
17.5k
Fork
1.9k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/open-gpu-kernel-modules

Tutte le issue di NVIDIA/open-gpu-kernel-modules

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.