Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug]: v26.7.x labels sandbox nodes with the `container` workload despite `sandboxWorkloads.defaultWorkload=vm-passthrough

Đang mở
  • Có gì không ổn?
#3,131 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@JunAr7112 đang làm issue này rồi.

Từ ngày 28/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

bug needs-triage

Describe the bug

With sandboxWorkloads.enabled=true and sandboxWorkloads.defaultWorkload=vm-passthrough, GPU Operator v26.7.x can label a GPU node without a nvidia.com/gpu.workload.config label with the container operand labels instead of the vm-passthrough ones. The VFIO manager and sandbox device plugin are then not deployed, and KubeVirt GPU passthrough fails. The wrong labels persist until the operator is restarted after a manual node label is added.

The operator log shows sandbox workloads enabled, but the default workload is still container:

WARNING: failed to get GPU workload config for node; using default
  NodeName=<node> SandboxEnabled=true Error="no GPU workload config found"
  defaultGPUWorkloadConfig=container

This configuration is documented as supported in GPU Operator with KubeVirt: "To override the default GPU workload configuration, set the following value in ClusterPolicy: sandboxWorkloads.defaultWorkload=<config>." It worked reliably for us with v25.3.1.

Analysis

In v26.7.0, node labeling moved from ClusterPolicyController.init() into the separate NodeLabelingReconciler. Both controllers react to ClusterPolicy creation and run concurrently, but:

  • NodeLabelingReconciler reads sandboxEnabled directly from the ClusterPolicy spec (controllers/nodelabeling_controller.go, updateGPUStateLabels).
  • For nodes without the workload label, it falls back to the package-level defaultGPUWorkloadConfig (controllers/state_manager.go, getWorkloadConfig). This variable is initialized to container.
  • That variable is updated from spec.sandboxWorkloads.defaultWorkload only in ClusterPolicyController.init(), which is called by ClusterPolicyReconciler after several API calls.

If node labeling runs first, it applies container operand labels. Nothing corrects this afterward:

  • The variable assignment is not a Kubernetes event.
  • Node update events are filtered by nodeUpdateRequiresReconcile.
  • ClusterPolicy update events are generation-only.
  • There is no periodic resync.

The same race recurs every time the operator restarts, because the variable is reset to container. That can also flip nodes that were previously labeled correctly.

Through v26.3.3, labelGPUNodes() ran inside ClusterPolicyController.init() after defaultGPUWorkloadConfig was assigned, so this ordering was guaranteed.

To Reproduce

  1. Use a single-node cluster with an NVIDIA GPU and NFD providing PCI labels, and no nvidia.com/gpu.workload.config node label.

  2. Install GPU Operator v26.7.2:

    sandboxWorkloads:
  enabled: true
  defaultWorkload: vm-passthrough
driver:
  enabled: false
toolkit:
  enabled: false

  3. Inspect the node labels and operator logs. On affected installs, the node receives nvidia.com/gpu.deploy.device-plugin=true and the other container operand labels instead of nvidia.com/gpu.deploy.vfio-manager=true and nvidia.com/gpu.deploy.sandbox-device-plugin=true. The log contains defaultGPUWorkloadConfig=container.

The issue is timing-dependent, so it may not reproduce on every install. Restarting the operator pod repeats the race.

Expected behavior

Nodes without nvidia.com/gpu.workload.config always use spec.sandboxWorkloads.defaultWorkload from the ClusterPolicy, independent of controller ordering or operator restarts.

A possible fix: in NodeLabelingReconciler, derive the fallback workload from cp.Spec.SandboxWorkloads.DefaultWorkload (when it is valid) instead of the shared package-level variable. The same applies to getNodeLabelUpdateReasons.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.2 (also present in v26.7.0 and v26.7.1 by code inspection; not present in v26.3.3 or earlier)
  • OS: SUSE Linux Micro 6.1
  • Kernel Version: 6.4.0 (SUSE)
  • Container Runtime Version: containerd v2.x (bundled with K3s)
  • Kubernetes Distro and Version: K3s v1.36.5+k3s1
  • Workload: KubeVirt VMs with GPU passthrough; NFD v0.20 installed separately (nfd.enabled=false)

Workaround

Explicitly label the node with nvidia.com/gpu.workload.config=vm-passthrough and restart the operator pod, or use v26.3.3 or earlier.

Ngôn ngữ chính
Go
Star
2.9k
Fork
587
Merge trung bình
2 ngày 2 giờ
Pull request đã merge (30 ngày)
62

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/gpu-operator

Tất cả issue của NVIDIA/gpu-operator

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.