[Bug]: 26.3.3 to 26.7.1 upgrade deletes an existing NVIDIADriver DaemonSet before driver-owner labels are initialized

未关闭
  • 哪里不对？
#3,155 1 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@rahulait 已经在做这个了。

开始于 2026年11月1日。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
14/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
go, helm, kubernetes
领域
infrastructure

调研方向

Start with cleanupStaleDriverDaemonsets and getManifestObjects in internal/state/driver.go, then getNodePools in internal/state/nodepool.go, which filters nodes by the driver-owner label. The question is why an existing owned DaemonSet is deleted when zero nodes match before the node-labeling controller in controllers/nodelabeling_controller.go assigns owners. Done means a DaemonSet is kept while owner labels are still pending, with a test covering the zero-node case.

由索引模型根据 Issue 内容生成。

描述

bug needs-triage

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug
During a GPU Operator 26.3.3 → 26.7.1 upgrade, the new operator selected zero nodes for an existing NVIDIADriver and deleted its driver DaemonSet. The GPU node then acquired nvidia.com/gpu-operator.driver.owner=default, and the same DaemonSet name was recreated with a different UID.

The installation already used NVIDIADriver/default before the upgrade. Its UID remained unchanged, its selector was empty, and it used compiled driver containers on both versions. The node remained present with the same OS. The owner label was absent under 26.3.3.

The following events were captured on 2026-10-31; all times are UTC:

Time Observation
21:52:45.514481300 Operator: No nodes matching the given node selector, driver default.
21:52:45.514501339 Operator: Deleting DaemonSet no longer managed by this CR.
21:52:45.531589 Watch: driver DaemonSet deleted, UID 92c0d471-b956-458d-bee1-4d94250d0627.
21:52:45.605381 Watch: GPU node first has nvidia.com/gpu-operator.driver.owner=default.
21:52:46.040006 Watch: same DaemonSet name added, UID 5faf12fd-2982-45eb-98b2-3a320c8f9c9b.

The DaemonSet name was nvidia-vgpu-driver-ubuntu24.04-86847f858c. Watch times are client receipt times; they are not API-server audit timestamps. The operator logs directly show that cleanup followed a zero-node match.

The source suggests a race between driver reconciliation and initial owner assignment:

  1. getNodePools requires the driver-owner label when selecting nodes.
  2. getManifestObjects returns an empty desired-object list when no nodes match.
  3. cleanupStaleDriverDaemonsets deletes the existing owned DaemonSet because its name is absent from that list.
  4. The separate node-labeling controller assigns owners, allowing the driver DaemonSet to be recreated.

To Reproduce

This was reproduced with independently reconciled operator and driver resources. An isolated Helm-only reproduction has not been run. The observed procedure was:

  1. Install GPU Operator 26.3.3 with one A30 GPU worker and an existing NVIDIADriver named default. Use driverType: vgpu, usePrecompiled: false, and an empty node selector. In this setup the GPU is exposed to the VM through PCI passthrough, while the driver resource uses the NVAIE guest-driver image.
  2. Start one Mistral 7B model replica using vLLM. Verify inference and GPU allocation. Record the driver DaemonSet UID and confirm that the GPU node lacks nvidia.com/gpu-operator.driver.owner.
  3. Capture operator logs, driver logs, and watches for nodes, NVIDIADriver resources, driver DaemonSets, and pods.
  4. Request operator 26.7.1 and driver 580.173.02. The baseline driver was 580.178.04. Allow the driver resource update to proceed without waiting for operator reconciliation to finish.
  5. The driver update also changes a custom environment variable from 26.3.3 to 26.7.1; its name is anonymized as REDACTED_CUSTOM_ENV_1 in the excerpts. Do not preassign the owner label or manually restart the driver.
  6. Observe the new operator's zero-node match and cleanup message. Compare the DaemonSet UID before and after the owner label first appears.

The driver resource's final spec.default is false; its name default does not indicate default-driver semantics. spec.manager is absent. Selected before/after resource fields and exact image digests are provided in driver-resources.json.

The old operator had already deleted the model pod through the full upgrade flow before this ownership transition. Two old-driver shutdown attempts failed to unload modules; the replacement driver manager later succeeded. The process retaining module references was not identified.

Expected behavior

The operator should preserve an existing driver DaemonSet while it initializes the owner labels needed to select its GPU nodes. A temporary absence of that operator-managed label should not, by itself, classify the existing DaemonSet as stale.

Driver changes should proceed through the controlled upgrade flow once ownership is established. This expectation does not require uninterrupted inference from a single GPU workload replica.

Environment (please provide the following information):

  • GPU Operator Version: 26.3.3 → 26.7.1.
  • OS: Ubuntu 24.04.5 LTS, amd64.
  • Kernel Version: 7.0.0-34-generic.
  • Container Runtime Version: containerd 2.2.7, FIPS build.
  • Kubernetes Distro and Version: Kubernetes 1.35.8; distribution name omitted.
主要语言
Go
星标
2.9k
派生
587
平均合并
1 天 22 小时
30 天内合并 PR
63

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/gpu-operator 的其他 Issue

查看 NVIDIA/gpu-operator 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。