[Bug]: 26.3.3 to 26.7.1 upgrade deletes an existing NVIDIADriver DaemonSet before driver-owner labels are initialized

Đang mở
  • Có gì không ổn?
#3,155 2 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@rahulait đang làm issue này rồi.

Từ ngày 1/11/2026.

  • #3157 của @rahulait — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
14/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go, helm, kubernetes
Lĩnh vực
infrastructure

Hướng nghiên cứu

Start with cleanupStaleDriverDaemonsets and getManifestObjects in internal/state/driver.go, then getNodePools in internal/state/nodepool.go, which filters nodes by the driver-owner label. The question is why an existing owned DaemonSet is deleted when zero nodes match before the node-labeling controller in controllers/nodelabeling_controller.go assigns owners. Done means a DaemonSet is kept while owner labels are still pending, with a test covering the zero-node case.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug
During a GPU Operator 26.3.3 → 26.7.1 upgrade, the new operator selected zero nodes for an existing NVIDIADriver and deleted its driver DaemonSet. The GPU node then acquired nvidia.com/gpu-operator.driver.owner=default, and the same DaemonSet name was recreated with a different UID.

The installation already used NVIDIADriver/default before the upgrade. Its UID remained unchanged, its selector was empty, and it used compiled driver containers on both versions. The node remained present with the same OS. The owner label was absent under 26.3.3.

The following events were captured on 2026-10-31; all times are UTC:

Time Observation
21:52:45.514481300 Operator: No nodes matching the given node selector, driver default.
21:52:45.514501339 Operator: Deleting DaemonSet no longer managed by this CR.
21:52:45.531589 Watch: driver DaemonSet deleted, UID 92c0d471-b956-458d-bee1-4d94250d0627.
21:52:45.605381 Watch: GPU node first has nvidia.com/gpu-operator.driver.owner=default.
21:52:46.040006 Watch: same DaemonSet name added, UID 5faf12fd-2982-45eb-98b2-3a320c8f9c9b.

The DaemonSet name was nvidia-vgpu-driver-ubuntu24.04-86847f858c. Watch times are client receipt times; they are not API-server audit timestamps. The operator logs directly show that cleanup followed a zero-node match.

The source suggests a race between driver reconciliation and initial owner assignment:

  1. getNodePools requires the driver-owner label when selecting nodes.
  2. getManifestObjects returns an empty desired-object list when no nodes match.
  3. cleanupStaleDriverDaemonsets deletes the existing owned DaemonSet because its name is absent from that list.
  4. The separate node-labeling controller assigns owners, allowing the driver DaemonSet to be recreated.

To Reproduce

This was reproduced with independently reconciled operator and driver resources. An isolated Helm-only reproduction has not been run. The observed procedure was:

  1. Install GPU Operator 26.3.3 with one A30 GPU worker and an existing NVIDIADriver named default. Use driverType: vgpu, usePrecompiled: false, and an empty node selector. In this setup the GPU is exposed to the VM through PCI passthrough, while the driver resource uses the NVAIE guest-driver image.
  2. Start one Mistral 7B model replica using vLLM. Verify inference and GPU allocation. Record the driver DaemonSet UID and confirm that the GPU node lacks nvidia.com/gpu-operator.driver.owner.
  3. Capture operator logs, driver logs, and watches for nodes, NVIDIADriver resources, driver DaemonSets, and pods.
  4. Request operator 26.7.1 and driver 580.173.02. The baseline driver was 580.178.04. Allow the driver resource update to proceed without waiting for operator reconciliation to finish.
  5. The driver update also changes a custom environment variable from 26.3.3 to 26.7.1; its name is anonymized as REDACTED_CUSTOM_ENV_1 in the excerpts. Do not preassign the owner label or manually restart the driver.
  6. Observe the new operator's zero-node match and cleanup message. Compare the DaemonSet UID before and after the owner label first appears.

The driver resource's final spec.default is false; its name default does not indicate default-driver semantics. spec.manager is absent. Selected before/after resource fields and exact image digests are provided in driver-resources.json.

The old operator had already deleted the model pod through the full upgrade flow before this ownership transition. Two old-driver shutdown attempts failed to unload modules; the replacement driver manager later succeeded. The process retaining module references was not identified.

Expected behavior

The operator should preserve an existing driver DaemonSet while it initializes the owner labels needed to select its GPU nodes. A temporary absence of that operator-managed label should not, by itself, classify the existing DaemonSet as stale.

Driver changes should proceed through the controlled upgrade flow once ownership is established. This expectation does not require uninterrupted inference from a single GPU workload replica.

Environment (please provide the following information):

  • GPU Operator Version: 26.3.3 → 26.7.1.
  • OS: Ubuntu 24.04.5 LTS, amd64.
  • Kernel Version: 7.0.0-34-generic.
  • Container Runtime Version: containerd 2.2.7, FIPS build.
  • Kubernetes Distro and Version: Kubernetes 1.35.8; distribution name omitted.
Ngôn ngữ chính
Go
Star
3k
Fork
592
Merge trung bình
2 ngày 22 giờ
Pull request đã merge (30 ngày)
62

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/gpu-operator

Tất cả issue của NVIDIA/gpu-operator

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.