NV_ERR_NO_MEMORY (0x51) at first CUDA context on 580.178.04 (RTX PRO 6000 Blackwell): LEVEL_ERROR on a successful 4K fallback?

Aberta
  • Algo errado?
#1,502 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
38/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
c, linux
Domínio
operating-systems

Direção de pesquisa

Start by checking the tagged 580.178.04 source paths named in the report: uvm_rm_mem.c, kern_gmmu.c, mem_desc.c, and system_mem.c. Trace the 2 MiB allocation request and fallback to 4 KiB, then compare the logged failure with the successful context creation described here. Done means confirming whether the error-level log is expected and whether fallback failure reaches a visible context creation error; the report does not identify a test.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Follow-up to #848 (closed with "open a new issue"). On 580.178.04 we see NV_ERR_NO_MEMORY at the first CUDA context of a process, apparently followed by a successful fallback. Our question is whether the LEVEL_ERROR print on a successful fallback can be demoted, and whether UVM could avoid asking for 2 MiB contiguous sysmem at GPU registration.

Environment

  • GPU: NVIDIA RTX PRO 6000 Blackwell (97887 MiB), one GPU per host, no MIG
  • Driver: 580.178.04, open kernel module
  • Kernel: 7.0.0-38-generic, x86_64
  • Persistence mode enabled (nvidia-persistenced active), memlock unlimited
  • THP: enabled=madvise, defrag=madvise
  • CUDA workloads run in containers with no cgroup memory limit

What we see

About 13-17 times per 24 h, at the first CUDA context creation of a container:

  • NVRM: Out of memory [NV_ERR_NO_MEMORY] (0x51) from _memdescAllocInternal (mem_desc.c)
  • the same status from sysmemConstruct (system_mem.c:356)
  • the same status from nvGpuOpsAllocPhysical
  • on some events also Allocation failed with big page size, retrying with default page size from _kgmmuClientShadowFaultBufferPagesAllocate

No Xid is logged and the GPU is not reset. In the cases we can join to the job outcome the job finishes normally with exit code 0.

Host memory is not exhausted at those moments (MemAvailable 79-90 GiB). It does look fragmented: /proc/vmstat shows compact_fail about 15.7 M vs compact_success about 1.7 M, and order-9/10 free blocks in /proc/buddyinfo drop to 0 at times. In our data the events cluster when a large memory-heavy process has just run (3 of 51 container starts during such runs, 0 of 59 outside them).

What we read in the source (tag 580.178.04)

This is our reading, please correct it if wrong.

  • uvm_rm_mem_alloc (uvm_rm_mem.c) asks for a 2M page size when size >= 2 MiB, else 64K, and retries with the default page size on NV_ERR_NO_MEMORY. That retry message goes through UVM_ERR_PRINT, so it is silent in release builds.
  • The request reaches nvGpuOpsAllocPhysical and then sysmemConstruct, which fails at LEVEL_ERROR for an explicit page size. This is the line that shows up in dmesg.
  • _kgmmuClientShadowFaultBufferPagesAllocate (kern_gmmu.c) tries RM_PAGE_SIZE_HUGE and retries at 4K, with a Bug 5042223 TODO.
  • uvm_rm_mem.c and kern_gmmu.c are identical between 580.178.04 and 580.196.07.

In #848 (2025-12-16) a collaborator wrote that a similar issue "was fixed in 580.xx ... where in cases of fragmented system memory, it would retry with smaller page sizes". If that retry is the intended fix, then the message looks like a normal fallback to us, but it is logged at error level, and health tooling that watches kernel logs cannot tell it apart from a real failure.

Questions

  1. Is the LEVEL_ERROR print on a successful fallback expected, and could it be demoted (or the fallback be made quiet) in a 580 branch release?
  2. Can UVM avoid requesting 2 MiB physically contiguous sysmem when a GPU is registered, since the 4 KiB path works?
  3. Is the Bug 5042223 TODO the tracker for this, and is there a driver version that changes the behaviour?
  4. If the 4 KiB fallback also fails, does context creation fail with a visible error? We have not been able to provoke that.

We can provide /proc/buddyinfo, /proc/vmstat and nvidia-bug-report.sh output if useful.

Linguagem predominante
C
Estrelas
17.6k
Forks
2k
Nenhum PR com merge em 30d

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de NVIDIA/open-gpu-kernel-modules

Todas as issues de NVIDIA/open-gpu-kernel-modules

Issues semelhantes

Mais issues de C

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.