[Bug]: Containers with NVIDIA_VISIBLE_DEVICES=all fail to start in jit-cdi mode when every GPU is bound to vfio-pci

Open Beginner friendly
  • Is something wrong?
#2,163 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@efegokdemir is already working on this.

Since Oct 27, 2026.

  • #2185 by @efegokdemir — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
tooling

Research direction

The bug is in pkg/nvcdi/lib-nvml.go where nvmllib.DeviceSpecGenerators calls nvmlInit before handling 'all'. Read that function and reorder so the 'all' case with no GPUs resolves to no devices like 'none'. Done means a container with NVIDIA_VISIBLE_DEVICES=all starts successfully when every GPU is bound to vfio-pci.

Written by the indexing model from the issue text.

Description

Describe the bug

When every GPU on a node is bound to vfio-pci (driver loaded, no GPU bound to it), the nvidia runtime in jit-cdi mode fails to create containers with NVIDIA_VISIBLE_DEVICES=all:

failed to create the automatic CDI modifier: failed to generate CDI spec for mode "auto": failed to construct device spec generators: failed to initialize NVML: No supported GPUs were found

This happens with the NVIDIA DRA driver's PassthroughSupport: while every GPU is in a KubeVirt VM, GPU Operator operands (gpu-feature-discovery, dcgm-exporter) cannot be recreated on that node.

nvmllib.DeviceSpecGenerators (pkg/nvcdi/lib-nvml.go) calls nvmlInit before handling all, and the init fails when no GPU is bound.

Expected behavior

all with no GPUs available resolves to no devices, as none does, and the container starts. I can send a PR.

Environment

  • nvidia-container-toolkit v1.19.1 (same code on main), driver 595.91.07
  • Talos v1.14.2, kernel 6.18.54, containerd 2.3.6, Kubernetes v1.37.1
  • RTX PRO 6000 Blackwell × 2, GPU Operator v26.7.1, DRA driver v0.5.1, KubeVirt v1.9.0
Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 12h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.