[Bug]: 26.3.3 to 26.7.1 upgrade deletes an existing NVIDIADriver DaemonSet before driver-owner labels are initialized

Open
  • Is something wrong?
#3,155 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@rahulait is already working on this.

Since Nov 1, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
14/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, helm, kubernetes
Domain
infrastructure

Research direction

Start with cleanupStaleDriverDaemonsets and getManifestObjects in internal/state/driver.go, then getNodePools in internal/state/nodepool.go, which filters nodes by the driver-owner label. The question is why an existing owned DaemonSet is deleted when zero nodes match before the node-labeling controller in controllers/nodelabeling_controller.go assigns owners. Done means a DaemonSet is kept while owner labels are still pending, with a test covering the zero-node case.

Written by the indexing model from the issue text.

Description

bug needs-triage

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug
During a GPU Operator 26.3.3 → 26.7.1 upgrade, the new operator selected zero nodes for an existing NVIDIADriver and deleted its driver DaemonSet. The GPU node then acquired nvidia.com/gpu-operator.driver.owner=default, and the same DaemonSet name was recreated with a different UID.

The installation already used NVIDIADriver/default before the upgrade. Its UID remained unchanged, its selector was empty, and it used compiled driver containers on both versions. The node remained present with the same OS. The owner label was absent under 26.3.3.

The following events were captured on 2026-10-31; all times are UTC:

Time Observation
21:52:45.514481300 Operator: No nodes matching the given node selector, driver default.
21:52:45.514501339 Operator: Deleting DaemonSet no longer managed by this CR.
21:52:45.531589 Watch: driver DaemonSet deleted, UID 92c0d471-b956-458d-bee1-4d94250d0627.
21:52:45.605381 Watch: GPU node first has nvidia.com/gpu-operator.driver.owner=default.
21:52:46.040006 Watch: same DaemonSet name added, UID 5faf12fd-2982-45eb-98b2-3a320c8f9c9b.

The DaemonSet name was nvidia-vgpu-driver-ubuntu24.04-86847f858c. Watch times are client receipt times; they are not API-server audit timestamps. The operator logs directly show that cleanup followed a zero-node match.

The source suggests a race between driver reconciliation and initial owner assignment:

  1. getNodePools requires the driver-owner label when selecting nodes.
  2. getManifestObjects returns an empty desired-object list when no nodes match.
  3. cleanupStaleDriverDaemonsets deletes the existing owned DaemonSet because its name is absent from that list.
  4. The separate node-labeling controller assigns owners, allowing the driver DaemonSet to be recreated.

To Reproduce

This was reproduced with independently reconciled operator and driver resources. An isolated Helm-only reproduction has not been run. The observed procedure was:

  1. Install GPU Operator 26.3.3 with one A30 GPU worker and an existing NVIDIADriver named default. Use driverType: vgpu, usePrecompiled: false, and an empty node selector. In this setup the GPU is exposed to the VM through PCI passthrough, while the driver resource uses the NVAIE guest-driver image.
  2. Start one Mistral 7B model replica using vLLM. Verify inference and GPU allocation. Record the driver DaemonSet UID and confirm that the GPU node lacks nvidia.com/gpu-operator.driver.owner.
  3. Capture operator logs, driver logs, and watches for nodes, NVIDIADriver resources, driver DaemonSets, and pods.
  4. Request operator 26.7.1 and driver 580.173.02. The baseline driver was 580.178.04. Allow the driver resource update to proceed without waiting for operator reconciliation to finish.
  5. The driver update also changes a custom environment variable from 26.3.3 to 26.7.1; its name is anonymized as REDACTED_CUSTOM_ENV_1 in the excerpts. Do not preassign the owner label or manually restart the driver.
  6. Observe the new operator's zero-node match and cleanup message. Compare the DaemonSet UID before and after the owner label first appears.

The driver resource's final spec.default is false; its name default does not indicate default-driver semantics. spec.manager is absent. Selected before/after resource fields and exact image digests are provided in driver-resources.json.

The old operator had already deleted the model pod through the full upgrade flow before this ownership transition. Two old-driver shutdown attempts failed to unload modules; the replacement driver manager later succeeded. The process retaining module references was not identified.

Expected behavior

The operator should preserve an existing driver DaemonSet while it initializes the owner labels needed to select its GPU nodes. A temporary absence of that operator-managed label should not, by itself, classify the existing DaemonSet as stale.

Driver changes should proceed through the controlled upgrade flow once ownership is established. This expectation does not require uninterrupted inference from a single GPU workload replica.

Environment (please provide the following information):

  • GPU Operator Version: 26.3.3 → 26.7.1.
  • OS: Ubuntu 24.04.5 LTS, amd64.
  • Kernel Version: 7.0.0-34-generic.
  • Container Runtime Version: containerd 2.2.7, FIPS build.
  • Kubernetes Distro and Version: Kubernetes 1.35.8; distribution name omitted.
Dominant language
Go
Stars
2.9k
Forks
587
Avg merge
1d 22h
Merged PRs (30d)
63

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/gpu-operator

All issues in NVIDIA/gpu-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.