[Bug]: default-kind management.nvidia.com/gpu incompatible with non-cdi strategies, making GPU pods unresolable
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 35/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- go, kubernetes
- Área
- infrastructure
Línea de trabajo
Start from the ClusterPolicy settings named in the report: DEVICE_LIST_STRATEGY on the device plugin, ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS on the toolkit, and the default-kind of management.nvidia.com/gpu that the operator sets. Reproduce with cdi.enabled=true and the volume-mounts strategy, then check that a bare device name is qualified with a kind that lacks it. Done means a workload pod starts with its GPU. The fix probably needs a maintainer decision on the default kind or on validation, so confirm the approach in the thread first.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Describe the bug
The operator sets default-kind to management.nvidia.com/gpu, and that kind contains only the "all" device by design. Any device list that arrives as bare names therefore cannot resolve in CDI mode: it is qualified with a kind that is guaranteed not to contain it. That makes the volume-mounts device list strategy and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS incompatible with cdi.enabled.
To Reproduce
- Deploy GPU Operator with cdi.enabled=true.
- Set
devicePlugin.env.DEVICE_LIST_STRATEGY=cdi-cri,cdi-annotations,volume-mountsandtoolkit.env.ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS=truein ClusterPolicy. - Run a pod requesting nvidia.com/gpu: 1 with
runtimeClassName: nvidia.
apiVersion: v1
kind: Pod
metadata:
name: test-cdi-device-plugin
namespace: cdi-test
spec:
tolerations:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
restartPolicy: Never
containers:
- name: gpu-test
image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
command: ["sh", "-c", "nvidia-smi -L"]
resources:
limits:
nvidia.com/gpu: 1
The container never starts with error -
Normal Pulling 12s kubelet spec.containers{gpu-test}: Pulling image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04"
Normal Pulled 8s kubelet spec.containers{gpu-test}: Successfully pulled image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04" in 4.301s (4.301s including waiting). Image size: 139357825 bytes.
Normal Created 8s kubelet spec.containers{gpu-test}: Container created
Warning Failed 8s kubelet spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-67fa9265-2b87-d8ce-e708-13e93416062c
Expected behavior
(1) The management.nvidia.com/gpu as default shouldn't get applies on workload pods.
(2) It would help if the CDI page stated that when CDI is enabled, DEVICE_LIST_STRATEGY must contain only CDI strategies and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS must be unset in ClusterPolicy - https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interfacehttps://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interface
Environment (please provide the following information):
GPU operator: v25.10
nvidia-container-toolkit version: v1.18.1
NVIDIA Driver Version: 580.126.20
Host OS: Ubuntu 24.04.4 LTS
Kernel Version: 6.8.0-1046-nvidia
Container Runtime Version: bundled with v1.35.2+rke2r1
CPU Architecture x86_64 and arm64
GPU Model(s): B200, H100
- Lenguaje dominante
- Go
- Estrellas
- 2.9k
- Forks
- 576
- Merge medio
- 1 d 5 h
- PR fusionados (30 d)
- 90
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de NVIDIA/gpu-operator
-
feature lifecycle/frozen needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
NVIDIA/gpu-operator#3074 ·
Los mantenedores suelen responder en 1 día
-
[Bug]: nvidia-validator waits the full timeout on a failed validation pod; loadPodSpec panicsAbiertobug needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
NVIDIA/gpu-operator#3065 ·
Los mantenedores suelen responder en 1 día
-
[Bug]: ClusterPolicy isDaemonSetReady can report Ready from a DaemonSet status of an older generationPosiblemente ocupada @efegokdemir la tomó hace 1 día. Abiertobug needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
NVIDIA/gpu-operator#3060 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
[Bug]: Driver upgrade does not evict pods that use nvidia.com/gpu only in a native sidecarPosiblemente ocupada @JunAr7112 la tomó hace 3 días. Abiertobug needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
NVIDIA/gpu-operator#3026 · 1 asignado ·
Los mantenedores suelen responder en 1 día
-
[Bug]: GPUCluster common name label breaks DRA validator selectorPosiblemente ocupada @ajavanma la tomó hace 22 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
NVIDIA/gpu-operator#2955 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de NVIDIA/gpu-operator
Issues similares
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Gentleman-Programming/engram#1899 ·
Los mantenedores suelen responder en 1 día
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
OpenTollGate/tollgate-module-basic-go#949 ·
Los mantenedores suelen responder en 1 día