Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: Helm chart always sets NVIDIADriver spec.manager.image, so a default driver-manager bump reinstalls the driver on every GPU node

未关闭
  • 哪里不对？
#3,135 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@akshat-lakhera 已经在做这个了。

开始于 2026年10月29日。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
go, helm, kubernetes
领域
devops, infrastructure

调研方向

Start with internal/state/driver.go and deployments/gpu-operator/templates/nvidiadriver.yaml, then check the driver.manager.image default in values.yaml and how DRIVER_CONFIG_DIGEST uses config.DriverManagerImage. Trace how the rendered manager image triggers the upgrade path, and determine how to avoid a reinstall when only the default manager version changes. Done means a chart upgrade with unchanged driver settings does not reinstall drivers across GPU nodes.

由索引模型根据 Issue 内容生成。

描述

Describe the bug

The Helm chart always sets NVIDIADriver.spec.manager.image, so an operator chart upgrade that only changes the default k8s-driver-manager version reinstalls the driver on every GPU node, even when the driver version and every user setting are unchanged.

internal/state/driver.go handleDefaultImagesInObjects is meant to avoid changing the manager image "when the operator itself is upgraded" while the NVIDIADriver spec is unmodified, but it only applies when cr.Spec.Manager.Image == "". deployments/gpu-operator/templates/nvidiadriver.yaml always renders manager: {{ toYaml .Values.driver.manager }}, and values.yaml sets driver.manager.image: k8s-driver-manager, so every Helm-managed NVIDIADriver picks up the new chart default (v0.12.0 in v26.7.0, v0.12.1 in v26.7.2). Because the manager image is part of DRIVER_CONFIG_DIGEST (config.DriverManagerImage), DriverPodRestartOnly does not apply, and every GPU node goes to upgrade-required and takes the full uninstall/reinstall flow.

In DRA mode this combines with the kubelet-plugin DaemonSet rolling in the same upgrade and deadlocks the first node taken, which NVIDIA/k8s-driver-manager#291 addresses.

To Reproduce

  1. Install gpu-operator v26.7.0 via the Helm chart with nvidiaDriver enabled (driver 595.91.07) and upgradePolicy.autoUpgrade: true, leaving driver.manager at its defaults.
  2. helm upgrade to v26.7.2 without changing any driver values.
  3. The NVIDIADriver's spec.manager.version changes from v0.12.0 to v0.12.1 and the driver DaemonSet template changes; every GPU node is labelled nvidia.com/gpu-driver-upgrade-state=upgrade-required and the upgrade controller cordons and restarts the driver on each node in turn, with the driver version unchanged.

Expected behavior

An operator chart upgrade that leaves the user's driver settings unchanged should not reinstall the driver: for example, do not render manager.image/version into the NVIDIADriver unless the user set them, so handleDefaultImagesInObjects applies, or exclude a default-only manager change from the reinstall path. Workaround: pin driver.manager.version in the Helm values.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.0 -> v26.7.2 (Helm)
  • OS: Ubuntu 24.04.5
  • Kernel Version: 6.17.0-42-generic
  • Container Runtime Version: containerd 2.2.1
  • Kubernetes Distro and Version: Cluster API on bare metal, Kubernetes v1.37.0, NVIDIA B200, DRA stack (GPUCluster) with maxParallelUpgrades: 1
主要语言
Go
星标
2.9k
派生
587
平均合并
2 天 2 小时
30 天内合并 PR
62

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/gpu-operator 的其他 Issue

查看 NVIDIA/gpu-operator 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。