[Bug]: nvidia-cdi-refresh.service fails at boot when NVreg_DeviceFileGID is non-zero: CapabilityBoundingSet lacks CAP_CHOWN

Aperta
  • Qualcosa non va?
#2,210 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@akhil9tiet ci sta già lavorando.

Dal 6/11/2026.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
72/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, linux
Ambito
devops, infrastructure

Direzione di ricerca

Start by reading the nvidia-cdi-refresh.service unit and the steps that create device nodes and generate the CDI spec; the issue identifies the restricted CapabilityBoundingSet as the likely cause. Reproduce with the supplied private /dev test and compare results with CAP_CHOWN added. Done means the service creates the device nodes and generates the CDI spec at boot with a non-zero NVreg_DeviceFileGID; the issue also suggests adding end-to-end coverage.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug needs-triage

Motivation

This is a real issue on our hosts. nvidia-cdi-refresh.service fails on every boot and gives up after hitting its start limit, so /var/run/cdi/nvidia.yaml is not generated until someone restarts the service manually. I think this is the root cause behind discussion #1873 (converted from #1611).

Check here if you would like to work on this issue

Describe the bug

On distributions whose driver packages set NVreg_DeviceFileGID to a non-zero group, nvidia-cdi-refresh.service cannot create the NVIDIA device nodes at boot. SUSE is one example: it ships options nvidia NVreg_DeviceFileGID=<video gid> NVreg_DeviceFileMode=0660 in /usr/lib/modprobe.d.

When /dev/nvidiactl and /dev/nvidia0 do not exist yet, NVML (through nvidia-smi -L or nvidia-ctk cdi generate) tries to create them and then chown them to the configured GID. The unit restricts CapabilityBoundingSet=CAP_SYS_MODULE CAP_SYS_ADMIN CAP_MKNOD, which does not include CAP_CHOWN, so node creation fails:

  • v1.20.x: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, then the service exits with status 127 because the last fallback path does not exist.
  • v1.18.x (no nvidia-smi step): failed to initialize NVML: Driver Not Loaded. This is the exact error reported in #1873.

The kernel module is already loaded at that point. In our boot logs, nvidia.ko loads about 0.6 s before the service starts.

The service only succeeds once some other full-privilege process has created the nodes. That explains the workarounds in #1873: After=nvidia-persistenced.service works because persistenced creates the nodes with full root capabilities, and adding a delay works by accident. On hosts where nvidia-persistenced is disabled, all 5 restart attempts fail and the unit stays failed.

The create-device-nodes step added in #1979/#2017 does not help:

  • It runs after nvidia-smi -L, so it is never reached when that step fails.
  • With the same bounding set it creates only the control nodes, not /dev/nvidia0, so nvidia-ctk cdi generate still fails.

The ordering drop-in from #2019 does not help either. In particular, K3s embeds containerd in k3s.service, so Before=containerd.service does not apply there.

To Reproduce

You can reproduce this on a running host without rebooting, by using an empty private /dev and the unit's bounding set:

cat > /tmp/repro.sh <<'EOF'
mount -t tmpfs none /dev
mknod -m 666 /dev/null c 1 3; mknod -m 666 /dev/zero c 1 5; mknod -m 666 /dev/urandom c 1 9
B=-all,+sys_module,+sys_admin,+mknod

echo "== unit bounding set:"
setpriv --bounding-set $B /usr/bin/nvidia-smi -L; echo "rc=$?"
setpriv --bounding-set $B /usr/bin/nvidia-ctk cdi generate --output=/tmp/t1.yaml 2>&1 | grep -E 'level=error'
rm -rf /dev/nvidia*

echo "== unit bounding set + CAP_CHOWN:"
setpriv --bounding-set $B,+chown /usr/bin/nvidia-smi -L; echo "rc=$?"
setpriv --bounding-set $B,+chown /usr/bin/nvidia-ctk cdi generate --output=/tmp/t2.yaml 2>&1 | grep -E 'Generated'
ls -l /dev | grep nvidia
EOF
sudo unshare -m --propagation private bash /tmp/repro.sh

Results:

Bounding set nvidia-smi -L nvidia-ctk cdi generate Device nodes
unit default couldn't communicate with the NVIDIA driver (rc=9) failed to initialize NVML: Driver Not Loaded not created
+ CAP_CHOWN lists the GPU (rc=0) Generated CDI spec created, group video
+ CAP_DAC_OVERRIDE / CAP_FOWNER / CAP_SETUID / CAP_SETGID / CAP_DAC_READ_SEARCH (each alone) fails — not created

At boot: with nvidia-persistenced disabled, the service fails 5 times and reports Start request repeated too quickly. After the first GPU container creates the nodes, systemctl restart nvidia-cdi-refresh.service succeeds.

Expected behavior

The service should create the device nodes and generate the CDI spec at boot without depending on another privileged component creating them first. Adding CAP_CHOWN to CapabilityBoundingSet fixes this in our testing. The following drop-in works as a workaround:

# /etc/systemd/system/nvidia-cdi-refresh.service.d/10-chown.conf
[Service]
CapabilityBoundingSet=CAP_CHOWN

It would also help to add e2e coverage with a non-zero NVreg_DeviceFileGID.

Environment (please provide the following information):

  • nvidia-container-toolkit version: v1.20.1 (the same root cause reproduces with v1.18-style nvidia-ctk cdi generate)
  • NVIDIA Driver Version: 580.178.04 (open kernel module, SUSE packages)
  • Host OS: SUSE Linux Micro 6.1
  • Kernel Version: 6.4.0 (SUSE default)
  • Container Runtime Version: containerd 2.3.4 (embedded in K3s)
  • CPU Architecture: x86_64
  • GPU Model(s): NVIDIA RTX PRO 2000 Blackwell

If applicable, also provide:

  • Kubernetes Distro and Version: K3s v1.36
  • NVIDIA GPU Operator version: not used on this host (DRA Driver for NVIDIA GPUs is used)
  • CUDA Version: N/A

Information to attach

Boot journal for nvidia-cdi-refresh.service:

systemd[1]: Starting Refresh NVIDIA CDI specification file...
sh[...]: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
sh[...]: /bin/sh: line 1: /usr/sbin/nvidia-smi: No such file or directory
sh[...]: /bin/sh: line 1: /usr/lib/wsl/lib/nvidia-smi: No such file or directory
systemd[1]: nvidia-cdi-refresh.service: Main process exited, code=exited, status=127/n/a
... (repeats 5 times)
systemd[1]: nvidia-cdi-refresh.service: Start request repeated too quickly.

Related: #1873, #1611, #1979, #2017, #2019, #2110

Lingua principale
Go
Stelle
4.6k
Fork
612
Merge medio
2g 14h
PR unite (30g)
41

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/nvidia-container-toolkit

Tutte le issue di NVIDIA/nvidia-container-toolkit

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.