Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug]: v26.7.x labels sandbox nodes with the `container` workload despite `sandboxWorkloads.defaultWorkload=vm-passthrough

未关闭
  • 哪里不对？
#3,131 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@JunAr7112 已经在做这个了。

开始于 2026年10月28日。

评估

这个 Issue 还没有评估数据。

描述

bug needs-triage

Describe the bug

With sandboxWorkloads.enabled=true and sandboxWorkloads.defaultWorkload=vm-passthrough, GPU Operator v26.7.x can label a GPU node without a nvidia.com/gpu.workload.config label with the container operand labels instead of the vm-passthrough ones. The VFIO manager and sandbox device plugin are then not deployed, and KubeVirt GPU passthrough fails. The wrong labels persist until the operator is restarted after a manual node label is added.

The operator log shows sandbox workloads enabled, but the default workload is still container:

WARNING: failed to get GPU workload config for node; using default
  NodeName=<node> SandboxEnabled=true Error="no GPU workload config found"
  defaultGPUWorkloadConfig=container

This configuration is documented as supported in GPU Operator with KubeVirt: "To override the default GPU workload configuration, set the following value in ClusterPolicy: sandboxWorkloads.defaultWorkload=<config>." It worked reliably for us with v25.3.1.

Analysis

In v26.7.0, node labeling moved from ClusterPolicyController.init() into the separate NodeLabelingReconciler. Both controllers react to ClusterPolicy creation and run concurrently, but:

  • NodeLabelingReconciler reads sandboxEnabled directly from the ClusterPolicy spec (controllers/nodelabeling_controller.go, updateGPUStateLabels).
  • For nodes without the workload label, it falls back to the package-level defaultGPUWorkloadConfig (controllers/state_manager.go, getWorkloadConfig). This variable is initialized to container.
  • That variable is updated from spec.sandboxWorkloads.defaultWorkload only in ClusterPolicyController.init(), which is called by ClusterPolicyReconciler after several API calls.

If node labeling runs first, it applies container operand labels. Nothing corrects this afterward:

  • The variable assignment is not a Kubernetes event.
  • Node update events are filtered by nodeUpdateRequiresReconcile.
  • ClusterPolicy update events are generation-only.
  • There is no periodic resync.

The same race recurs every time the operator restarts, because the variable is reset to container. That can also flip nodes that were previously labeled correctly.

Through v26.3.3, labelGPUNodes() ran inside ClusterPolicyController.init() after defaultGPUWorkloadConfig was assigned, so this ordering was guaranteed.

To Reproduce

  1. Use a single-node cluster with an NVIDIA GPU and NFD providing PCI labels, and no nvidia.com/gpu.workload.config node label.

  2. Install GPU Operator v26.7.2:

    sandboxWorkloads:
  enabled: true
  defaultWorkload: vm-passthrough
driver:
  enabled: false
toolkit:
  enabled: false

  3. Inspect the node labels and operator logs. On affected installs, the node receives nvidia.com/gpu.deploy.device-plugin=true and the other container operand labels instead of nvidia.com/gpu.deploy.vfio-manager=true and nvidia.com/gpu.deploy.sandbox-device-plugin=true. The log contains defaultGPUWorkloadConfig=container.

The issue is timing-dependent, so it may not reproduce on every install. Restarting the operator pod repeats the race.

Expected behavior

Nodes without nvidia.com/gpu.workload.config always use spec.sandboxWorkloads.defaultWorkload from the ClusterPolicy, independent of controller ordering or operator restarts.

A possible fix: in NodeLabelingReconciler, derive the fallback workload from cp.Spec.SandboxWorkloads.DefaultWorkload (when it is valid) instead of the shared package-level variable. The same applies to getNodeLabelUpdateReasons.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.2 (also present in v26.7.0 and v26.7.1 by code inspection; not present in v26.3.3 or earlier)
  • OS: SUSE Linux Micro 6.1
  • Kernel Version: 6.4.0 (SUSE)
  • Container Runtime Version: containerd v2.x (bundled with K3s)
  • Kubernetes Distro and Version: K3s v1.36.5+k3s1
  • Workload: KubeVirt VMs with GPU passthrough; NFD v0.20 installed separately (nfd.enabled=false)

Workaround

Explicitly label the node with nvidia.com/gpu.workload.config=vm-passthrough and restart the operator pod, or use v26.3.3 or earlier.

主要语言
Go
星标
2.9k
派生
587
平均合并
2 天 2 小时
30 天内合并 PR
62

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/gpu-operator 的其他 Issue

查看 NVIDIA/gpu-operator 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。