[Bug]: Helm chart always sets NVIDIADriver spec.manager.image, so a default driver-manager bump reinstalls the driver on every GPU node

Abierto
  • ¿Algo no está bien?
#3,135 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@akshat-lakhera ya está trabajando en esto.

Desde el 29/10/2026.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, helm, kubernetes
Área
devops, infrastructure

Línea de trabajo

Start with internal/state/driver.go and deployments/gpu-operator/templates/nvidiadriver.yaml, then check the driver.manager.image default in values.yaml and how DRIVER_CONFIG_DIGEST uses config.DriverManagerImage. Trace how the rendered manager image triggers the upgrade path, and determine how to avoid a reinstall when only the default manager version changes. Done means a chart upgrade with unchanged driver settings does not reinstall drivers across GPU nodes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

The Helm chart always sets NVIDIADriver.spec.manager.image, so an operator chart upgrade that only changes the default k8s-driver-manager version reinstalls the driver on every GPU node, even when the driver version and every user setting are unchanged.

internal/state/driver.go handleDefaultImagesInObjects is meant to avoid changing the manager image "when the operator itself is upgraded" while the NVIDIADriver spec is unmodified, but it only applies when cr.Spec.Manager.Image == "". deployments/gpu-operator/templates/nvidiadriver.yaml always renders manager: {{ toYaml .Values.driver.manager }}, and values.yaml sets driver.manager.image: k8s-driver-manager, so every Helm-managed NVIDIADriver picks up the new chart default (v0.12.0 in v26.7.0, v0.12.1 in v26.7.2). Because the manager image is part of DRIVER_CONFIG_DIGEST (config.DriverManagerImage), DriverPodRestartOnly does not apply, and every GPU node goes to upgrade-required and takes the full uninstall/reinstall flow.

In DRA mode this combines with the kubelet-plugin DaemonSet rolling in the same upgrade and deadlocks the first node taken, which NVIDIA/k8s-driver-manager#291 addresses.

To Reproduce

  1. Install gpu-operator v26.7.0 via the Helm chart with nvidiaDriver enabled (driver 595.91.07) and upgradePolicy.autoUpgrade: true, leaving driver.manager at its defaults.
  2. helm upgrade to v26.7.2 without changing any driver values.
  3. The NVIDIADriver's spec.manager.version changes from v0.12.0 to v0.12.1 and the driver DaemonSet template changes; every GPU node is labelled nvidia.com/gpu-driver-upgrade-state=upgrade-required and the upgrade controller cordons and restarts the driver on each node in turn, with the driver version unchanged.

Expected behavior

An operator chart upgrade that leaves the user's driver settings unchanged should not reinstall the driver: for example, do not render manager.image/version into the NVIDIADriver unless the user set them, so handleDefaultImagesInObjects applies, or exclude a default-only manager change from the reinstall path. Workaround: pin driver.manager.version in the Helm values.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.0 -> v26.7.2 (Helm)
  • OS: Ubuntu 24.04.5
  • Kernel Version: 6.17.0-42-generic
  • Container Runtime Version: containerd 2.2.1
  • Kubernetes Distro and Version: Cluster API on bare metal, Kubernetes v1.37.0, NVIDIA B200, DRA stack (GPUCluster) with maxParallelUpgrades: 1
Lenguaje dominante
Go
Estrellas
2.9k
Forks
587
Merge medio
2 d 8 h
PR fusionados (30 d)
64

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/gpu-operator

Todos los issues de NVIDIA/gpu-operator

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.