[Bug]: v26.7.x labels sandbox nodes with the `container` workload despite `sandboxWorkloads.defaultWorkload=vm-passthrough

Abierto
  • ¿Algo no está bien?
#3,131 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@JunAr7112 ya está trabajando en esto.

Desde el 28/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

bug needs-triage

Describe the bug

With sandboxWorkloads.enabled=true and sandboxWorkloads.defaultWorkload=vm-passthrough, GPU Operator v26.7.x can label a GPU node without a nvidia.com/gpu.workload.config label with the container operand labels instead of the vm-passthrough ones. The VFIO manager and sandbox device plugin are then not deployed, and KubeVirt GPU passthrough fails. The wrong labels persist until the operator is restarted after a manual node label is added.

The operator log shows sandbox workloads enabled, but the default workload is still container:

WARNING: failed to get GPU workload config for node; using default
  NodeName=<node> SandboxEnabled=true Error="no GPU workload config found"
  defaultGPUWorkloadConfig=container

This configuration is documented as supported in GPU Operator with KubeVirt: "To override the default GPU workload configuration, set the following value in ClusterPolicy: sandboxWorkloads.defaultWorkload=<config>." It worked reliably for us with v25.3.1.

Analysis

In v26.7.0, node labeling moved from ClusterPolicyController.init() into the separate NodeLabelingReconciler. Both controllers react to ClusterPolicy creation and run concurrently, but:

  • NodeLabelingReconciler reads sandboxEnabled directly from the ClusterPolicy spec (controllers/nodelabeling_controller.go, updateGPUStateLabels).
  • For nodes without the workload label, it falls back to the package-level defaultGPUWorkloadConfig (controllers/state_manager.go, getWorkloadConfig). This variable is initialized to container.
  • That variable is updated from spec.sandboxWorkloads.defaultWorkload only in ClusterPolicyController.init(), which is called by ClusterPolicyReconciler after several API calls.

If node labeling runs first, it applies container operand labels. Nothing corrects this afterward:

  • The variable assignment is not a Kubernetes event.
  • Node update events are filtered by nodeUpdateRequiresReconcile.
  • ClusterPolicy update events are generation-only.
  • There is no periodic resync.

The same race recurs every time the operator restarts, because the variable is reset to container. That can also flip nodes that were previously labeled correctly.

Through v26.3.3, labelGPUNodes() ran inside ClusterPolicyController.init() after defaultGPUWorkloadConfig was assigned, so this ordering was guaranteed.

To Reproduce

  1. Use a single-node cluster with an NVIDIA GPU and NFD providing PCI labels, and no nvidia.com/gpu.workload.config node label.

  2. Install GPU Operator v26.7.2:

    sandboxWorkloads:
  enabled: true
  defaultWorkload: vm-passthrough
driver:
  enabled: false
toolkit:
  enabled: false

  3. Inspect the node labels and operator logs. On affected installs, the node receives nvidia.com/gpu.deploy.device-plugin=true and the other container operand labels instead of nvidia.com/gpu.deploy.vfio-manager=true and nvidia.com/gpu.deploy.sandbox-device-plugin=true. The log contains defaultGPUWorkloadConfig=container.

The issue is timing-dependent, so it may not reproduce on every install. Restarting the operator pod repeats the race.

Expected behavior

Nodes without nvidia.com/gpu.workload.config always use spec.sandboxWorkloads.defaultWorkload from the ClusterPolicy, independent of controller ordering or operator restarts.

A possible fix: in NodeLabelingReconciler, derive the fallback workload from cp.Spec.SandboxWorkloads.DefaultWorkload (when it is valid) instead of the shared package-level variable. The same applies to getNodeLabelUpdateReasons.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.2 (also present in v26.7.0 and v26.7.1 by code inspection; not present in v26.3.3 or earlier)
  • OS: SUSE Linux Micro 6.1
  • Kernel Version: 6.4.0 (SUSE)
  • Container Runtime Version: containerd v2.x (bundled with K3s)
  • Kubernetes Distro and Version: K3s v1.36.5+k3s1
  • Workload: KubeVirt VMs with GPU passthrough; NFD v0.20 installed separately (nfd.enabled=false)

Workaround

Explicitly label the node with nvidia.com/gpu.workload.config=vm-passthrough and restart the operator pod, or use v26.3.3 or earlier.

Lenguaje dominante
Go
Estrellas
2.9k
Forks
587
Merge medio
2 d 2 h
PR fusionados (30 d)
62

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/gpu-operator

Todos los issues de NVIDIA/gpu-operator

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.