Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: default-kind management.nvidia.com/gpu incompatible with non-cdi strategies, making GPU pods unresolable

Aperta
  • Qualcosa non va?
#3,077 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, kubernetes
Ambito
infrastructure

Direzione di ricerca

Start from the ClusterPolicy settings named in the report: DEVICE_LIST_STRATEGY on the device plugin, ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS on the toolkit, and the default-kind of management.nvidia.com/gpu that the operator sets. Reproduce with cdi.enabled=true and the volume-mounts strategy, then check that a bare device name is qualified with a kind that lacks it. Done means a workload pod starts with its GPU. The fix probably needs a maintainer decision on the default kind or on validation, so confirm the approach in the thread first.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug needs-triage

Describe the bug
The operator sets default-kind to management.nvidia.com/gpu, and that kind contains only the "all" device by design. Any device list that arrives as bare names therefore cannot resolve in CDI mode: it is qualified with a kind that is guaranteed not to contain it. That makes the volume-mounts device list strategy and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS incompatible with cdi.enabled.

To Reproduce

  1. Deploy GPU Operator with cdi.enabled=true.
  2. Set devicePlugin.env.DEVICE_LIST_STRATEGY=cdi-cri,cdi-annotations,volume-mounts and toolkit.env.ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS=true in ClusterPolicy.
  3. Run a pod requesting nvidia.com/gpu: 1 with runtimeClassName: nvidia.
  apiVersion: v1
  kind: Pod
  metadata:
    name: test-cdi-device-plugin
    namespace: cdi-test
  spec:
    tolerations:
    - key: nvidia.com/gpu
      value: "true"
      effect: NoSchedule
    restartPolicy: Never
    containers:
    - name: gpu-test
      image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
      command: ["sh", "-c", "nvidia-smi -L"]
      resources:
        limits:
          nvidia.com/gpu: 1

The container never starts with error -

  Normal   Pulling    12s   kubelet            spec.containers{gpu-test}: Pulling image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04"
  Normal   Pulled     8s    kubelet            spec.containers{gpu-test}: Successfully pulled image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04" in 4.301s (4.301s including waiting). Image size: 139357825 bytes.
  Normal   Created    8s    kubelet            spec.containers{gpu-test}: Container created
  Warning  Failed     8s    kubelet            spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-67fa9265-2b87-d8ce-e708-13e93416062c

Expected behavior
(1) The management.nvidia.com/gpu as default shouldn't get applies on workload pods.

(2) It would help if the CDI page stated that when CDI is enabled, DEVICE_LIST_STRATEGY must contain only CDI strategies and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS must be unset in ClusterPolicy - https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interfacehttps://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interface

Environment (please provide the following information):
GPU operator: v25.10
nvidia-container-toolkit version: v1.18.1
NVIDIA Driver Version: 580.126.20
Host OS: Ubuntu 24.04.4 LTS
Kernel Version: 6.8.0-1046-nvidia
Container Runtime Version: bundled with v1.35.2+rke2r1
CPU Architecture x86_64 and arm64
GPU Model(s): B200, H100

Lingua principale
Go
Stelle
2.9k
Fork
569
Merge medio
1g 5h
PR unite (30g)
90

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/gpu-operator

Tutte le issue di NVIDIA/gpu-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.