Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug]: v26.7.x labels sandbox nodes with the `container` workload despite `sandboxWorkloads.defaultWorkload=vm-passthrough

オープン
  • 何か問題がありますか？
#3,131 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@JunAr7112 がすでに取り組んでいます。

2026年10月28日 から。

評価

この issue はまだ評価されていません。

説明

bug needs-triage

Describe the bug

With sandboxWorkloads.enabled=true and sandboxWorkloads.defaultWorkload=vm-passthrough, GPU Operator v26.7.x can label a GPU node without a nvidia.com/gpu.workload.config label with the container operand labels instead of the vm-passthrough ones. The VFIO manager and sandbox device plugin are then not deployed, and KubeVirt GPU passthrough fails. The wrong labels persist until the operator is restarted after a manual node label is added.

The operator log shows sandbox workloads enabled, but the default workload is still container:

WARNING: failed to get GPU workload config for node; using default
  NodeName=<node> SandboxEnabled=true Error="no GPU workload config found"
  defaultGPUWorkloadConfig=container

This configuration is documented as supported in GPU Operator with KubeVirt: "To override the default GPU workload configuration, set the following value in ClusterPolicy: sandboxWorkloads.defaultWorkload=<config>." It worked reliably for us with v25.3.1.

Analysis

In v26.7.0, node labeling moved from ClusterPolicyController.init() into the separate NodeLabelingReconciler. Both controllers react to ClusterPolicy creation and run concurrently, but:

  • NodeLabelingReconciler reads sandboxEnabled directly from the ClusterPolicy spec (controllers/nodelabeling_controller.go, updateGPUStateLabels).
  • For nodes without the workload label, it falls back to the package-level defaultGPUWorkloadConfig (controllers/state_manager.go, getWorkloadConfig). This variable is initialized to container.
  • That variable is updated from spec.sandboxWorkloads.defaultWorkload only in ClusterPolicyController.init(), which is called by ClusterPolicyReconciler after several API calls.

If node labeling runs first, it applies container operand labels. Nothing corrects this afterward:

  • The variable assignment is not a Kubernetes event.
  • Node update events are filtered by nodeUpdateRequiresReconcile.
  • ClusterPolicy update events are generation-only.
  • There is no periodic resync.

The same race recurs every time the operator restarts, because the variable is reset to container. That can also flip nodes that were previously labeled correctly.

Through v26.3.3, labelGPUNodes() ran inside ClusterPolicyController.init() after defaultGPUWorkloadConfig was assigned, so this ordering was guaranteed.

To Reproduce

  1. Use a single-node cluster with an NVIDIA GPU and NFD providing PCI labels, and no nvidia.com/gpu.workload.config node label.

  2. Install GPU Operator v26.7.2:

    sandboxWorkloads:
  enabled: true
  defaultWorkload: vm-passthrough
driver:
  enabled: false
toolkit:
  enabled: false

  3. Inspect the node labels and operator logs. On affected installs, the node receives nvidia.com/gpu.deploy.device-plugin=true and the other container operand labels instead of nvidia.com/gpu.deploy.vfio-manager=true and nvidia.com/gpu.deploy.sandbox-device-plugin=true. The log contains defaultGPUWorkloadConfig=container.

The issue is timing-dependent, so it may not reproduce on every install. Restarting the operator pod repeats the race.

Expected behavior

Nodes without nvidia.com/gpu.workload.config always use spec.sandboxWorkloads.defaultWorkload from the ClusterPolicy, independent of controller ordering or operator restarts.

A possible fix: in NodeLabelingReconciler, derive the fallback workload from cp.Spec.SandboxWorkloads.DefaultWorkload (when it is valid) instead of the shared package-level variable. The same applies to getNodeLabelUpdateReasons.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.2 (also present in v26.7.0 and v26.7.1 by code inspection; not present in v26.3.3 or earlier)
  • OS: SUSE Linux Micro 6.1
  • Kernel Version: 6.4.0 (SUSE)
  • Container Runtime Version: containerd v2.x (bundled with K3s)
  • Kubernetes Distro and Version: K3s v1.36.5+k3s1
  • Workload: KubeVirt VMs with GPU passthrough; NFD v0.20 installed separately (nfd.enabled=false)

Workaround

Explicitly label the node with nvidia.com/gpu.workload.config=vm-passthrough and restart the operator pod, or use v26.3.3 or earlier.

主要言語
Go
スター
2.9k
フォーク
587
平均マージ
2日 2時間
マージ済み PR（30日）
62

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/gpu-operator のほかの issue

NVIDIA/gpu-operator の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。