Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: GPUCluster DRA kubelet plugin uses driver root / on Talos and cannot find libnvidia-ml.so.1

Aperta
  • Qualcosa non va?
#3,139 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@roma5087 ci sta già lavorando.

Dal 29/10/2026.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, kubernetes
Ambito
infrastructure

Direzione di ricerca

Start by tracing runValidation, resolveHostNvidiaSMI, and getDriverInfo, then inspect manifests/state-dra-driver/0500_daemonset.yaml and how hostPaths.driverInstallDir is passed into the validation flow. Reproduce the Talos case with the driver under /usr/local and check the generated driver-ready values. Done when the DRA kubelet plugin finds libnvidia-ml.so.1 and publishes its ResourceSlice using the configured driver root.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

On Talos Linux the DRA kubelet plugin deployed through a GPUCluster crashes with error locating "libnvidia-ml.so.1", so no ResourceSlice is published and DCGM Exporter and the DRA validation stay Pending.

Talos ships the driver as a system extension under /usr/local, and /usr/bin/nvidia-smi is a link to /usr/local/bin/nvidia-smi. The validator's runValidation tries the host driver first: resolveHostNvidiaSMI finds /usr/bin/nvidia-smi, chroot /host /usr/bin/nvidia-smi succeeds, and getDriverInfo(true, ...) sets the driver root to the host root. driver-ready then contains NVIDIA_DRIVER_ROOT=/ and hostPaths.driverInstallDir: /usr/local from the GPUCluster is never used. The kubelet plugin sources driver-ready (manifests/state-dra-driver/0500_daemonset.yaml), looks for libnvidia-ml.so.1 below / and does not find it in /usr/local/lib.

I found no field in GPUCluster to set the driver root. Setting it through kubeletPlugin.env has no effect, because the container sources driver-ready afterwards and overwrites it.

As a check I patched NVIDIA_DRIVER_ROOT=/usr/local into the kubelet plugin DaemonSet by hand. With that the GPUCluster becomes ready and DCGM Exporter labels the metrics with the pod holding the DRA claim, so the rest of the path works on Talos:

DCGM_FI_DEV_GPU_UTIL{gpu="0",...,container="cuda",dra_claim_name="dra-hold-gpu-hj5r2",dra_claim_namespace="default",dra_device_name="gpu-0",dra_driver_name="gpu.nvidia.com",dra_pool_name="gpu",namespace="default",pod="dra-hold"} 0

The standalone DRA driver chart works on the same node with nvidiaDriverRoot: /usr/local.

To Reproduce

  1. Talos node with the NVIDIA system extensions (driver under /usr/local).
  2. Install the GPU Operator v26.7.1 with driver and toolkit disabled, as the Talos guide describes, and create a GPUCluster with the DRA kubelet plugin enabled and hostPaths.driverInstallDir: /usr/local.
  3. The kubelet plugin pod crash-loops with error locating "libnvidia-ml.so.1"; /run/nvidia/validations/driver-ready on the node contains IS_HOST_DRIVER=true and NVIDIA_DRIVER_ROOT=/.

Expected behavior

The driver root the plugin uses is /usr/local, either because a host driver found under a non-standard prefix is reported with that prefix, or because an explicit hostPaths.driverInstallDir (or a dedicated field) wins over the host-driver detection.

Environment

  • GPU Operator Version: v26.7.1 (I read v26.7.2 and main and see the same logic, not tested live)
  • DRA driver: v0.5.1, DCGM Exporter 4.6.1-4.8.4-distroless
  • OS: Talos Linux v1.14.5 with the NVIDIA proprietary driver extensions
  • Container Runtime Version: containerd as shipped with Talos v1.14.5
  • Kubernetes Distro and Version: upstream Kubernetes 1.37.1
  • GPU: RTX A1000

Related: #2731

Lingua principale
Go
Stelle
2.9k
Fork
587
Merge medio
2g 2h
PR unite (30g)
62

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/gpu-operator

Tutte le issue di NVIDIA/gpu-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.