[Bug]: default-kind management.nvidia.com/gpu incompatible with non-cdi strategies, making GPU pods unresolable
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 1-2 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- go, kubernetes
调研方向
Start from the ClusterPolicy settings named in the report: DEVICE_LIST_STRATEGY on the device plugin, ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS on the toolkit, and the default-kind of management.nvidia.com/gpu that the operator sets. Reproduce with cdi.enabled=true and the volume-mounts strategy, then check that a bare device name is qualified with a kind that lacks it. Done means a workload pod starts with its GPU. The fix probably needs a maintainer decision on the default kind or on validation, so confirm the approach in the thread first.
由索引模型根据 Issue 内容生成。
描述
Describe the bug
The operator sets default-kind to management.nvidia.com/gpu, and that kind contains only the "all" device by design. Any device list that arrives as bare names therefore cannot resolve in CDI mode: it is qualified with a kind that is guaranteed not to contain it. That makes the volume-mounts device list strategy and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS incompatible with cdi.enabled.
To Reproduce
- Deploy GPU Operator with cdi.enabled=true.
- Set
devicePlugin.env.DEVICE_LIST_STRATEGY=cdi-cri,cdi-annotations,volume-mountsandtoolkit.env.ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS=truein ClusterPolicy. - Run a pod requesting nvidia.com/gpu: 1 with
runtimeClassName: nvidia.
apiVersion: v1
kind: Pod
metadata:
name: test-cdi-device-plugin
namespace: cdi-test
spec:
tolerations:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
restartPolicy: Never
containers:
- name: gpu-test
image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
command: ["sh", "-c", "nvidia-smi -L"]
resources:
limits:
nvidia.com/gpu: 1
The container never starts with error -
Normal Pulling 12s kubelet spec.containers{gpu-test}: Pulling image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04"
Normal Pulled 8s kubelet spec.containers{gpu-test}: Successfully pulled image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04" in 4.301s (4.301s including waiting). Image size: 139357825 bytes.
Normal Created 8s kubelet spec.containers{gpu-test}: Container created
Warning Failed 8s kubelet spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-67fa9265-2b87-d8ce-e708-13e93416062c
Expected behavior
(1) The management.nvidia.com/gpu as default shouldn't get applies on workload pods.
(2) It would help if the CDI page stated that when CDI is enabled, DEVICE_LIST_STRATEGY must contain only CDI strategies and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS must be unset in ClusterPolicy - https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interfacehttps://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interface
Environment (please provide the following information):
GPU operator: v25.10
nvidia-container-toolkit version: v1.18.1
NVIDIA Driver Version: 580.126.20
Host OS: Ubuntu 24.04.4 LTS
Kernel Version: 6.8.0-1046-nvidia
Container Runtime Version: bundled with v1.35.2+rke2r1
CPU Architecture x86_64 and arm64
GPU Model(s): B200, H100
- 主要语言
- Go
- 星标
- 2.9k
- 派生
- 569
- 平均合并
- 1 天 5 小时
- 30 天内合并 PR
- 90
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
NVIDIA/gpu-operator 的其他 Issue
-
feature lifecycle/frozen needs-triage
难度 2/5 1-3 小时 新手友好度 68/100
NVIDIA/gpu-operator#3074 ·
维护者通常 1 天内回复
-
bug needs-triage
难度 2/5 1-3 小时 新手友好度 75/100
NVIDIA/gpu-operator#3065 ·
维护者通常 1 天内回复
-
[Bug]: ClusterPolicy isDaemonSetReady can report Ready from a DaemonSet status of an older generation可能已有人在做 @efegokdemir 今天认领。 未关闭bug needs-triage
难度 2/5 1-3 小时 新手友好度 72/100
NVIDIA/gpu-operator#3060 · 1 条评论 ·
维护者通常 1 天内回复
-
[Bug]: Driver upgrade does not evict pods that use nvidia.com/gpu only in a native sidecar可能已有人在做 @JunAr7112 于 2 天前认领。 未关闭bug needs-triage
难度 2/5 1-3 小时 新手友好度 78/100
NVIDIA/gpu-operator#3026 · 已指派 1 人 ·
维护者通常 1 天内回复
-
[Bug]: GPUCluster common name label breaks DRA validator selector可能已有人在做 @ajavanma 于 22 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 88/100
NVIDIA/gpu-operator#2955 · 1 条评论 ·
维护者通常 1 天内回复
查看 NVIDIA/gpu-operator 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
oalders/clodhopper#162 ·
维护者通常 1 天内回复
-
component/server severity/minor type/bug
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
diagnose globalnet reports a failure for every exported headless Service可能已有人在做 @kurtyoon 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
submariner-io/subctl#1948 ·
维护者通常 1 天内回复