Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug]: GPUCluster DRA kubelet plugin uses driver root / on Talos and cannot find libnvidia-ml.so.1

オープン
  • 何か問題がありますか？
#3,139 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@roma5087 がすでに取り組んでいます。

2026年10月29日 から。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
52/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
go, kubernetes
領域
infrastructure

調査の方向性

Start by tracing runValidation, resolveHostNvidiaSMI, and getDriverInfo, then inspect manifests/state-dra-driver/0500_daemonset.yaml and how hostPaths.driverInstallDir is passed into the validation flow. Reproduce the Talos case with the driver under /usr/local and check the generated driver-ready values. Done when the DRA kubelet plugin finds libnvidia-ml.so.1 and publishes its ResourceSlice using the configured driver root.

索引モデルが issue の本文から書いたものです。

説明

Describe the bug

On Talos Linux the DRA kubelet plugin deployed through a GPUCluster crashes with error locating "libnvidia-ml.so.1", so no ResourceSlice is published and DCGM Exporter and the DRA validation stay Pending.

Talos ships the driver as a system extension under /usr/local, and /usr/bin/nvidia-smi is a link to /usr/local/bin/nvidia-smi. The validator's runValidation tries the host driver first: resolveHostNvidiaSMI finds /usr/bin/nvidia-smi, chroot /host /usr/bin/nvidia-smi succeeds, and getDriverInfo(true, ...) sets the driver root to the host root. driver-ready then contains NVIDIA_DRIVER_ROOT=/ and hostPaths.driverInstallDir: /usr/local from the GPUCluster is never used. The kubelet plugin sources driver-ready (manifests/state-dra-driver/0500_daemonset.yaml), looks for libnvidia-ml.so.1 below / and does not find it in /usr/local/lib.

I found no field in GPUCluster to set the driver root. Setting it through kubeletPlugin.env has no effect, because the container sources driver-ready afterwards and overwrites it.

As a check I patched NVIDIA_DRIVER_ROOT=/usr/local into the kubelet plugin DaemonSet by hand. With that the GPUCluster becomes ready and DCGM Exporter labels the metrics with the pod holding the DRA claim, so the rest of the path works on Talos:

DCGM_FI_DEV_GPU_UTIL{gpu="0",...,container="cuda",dra_claim_name="dra-hold-gpu-hj5r2",dra_claim_namespace="default",dra_device_name="gpu-0",dra_driver_name="gpu.nvidia.com",dra_pool_name="gpu",namespace="default",pod="dra-hold"} 0

The standalone DRA driver chart works on the same node with nvidiaDriverRoot: /usr/local.

To Reproduce

  1. Talos node with the NVIDIA system extensions (driver under /usr/local).
  2. Install the GPU Operator v26.7.1 with driver and toolkit disabled, as the Talos guide describes, and create a GPUCluster with the DRA kubelet plugin enabled and hostPaths.driverInstallDir: /usr/local.
  3. The kubelet plugin pod crash-loops with error locating "libnvidia-ml.so.1"; /run/nvidia/validations/driver-ready on the node contains IS_HOST_DRIVER=true and NVIDIA_DRIVER_ROOT=/.

Expected behavior

The driver root the plugin uses is /usr/local, either because a host driver found under a non-standard prefix is reported with that prefix, or because an explicit hostPaths.driverInstallDir (or a dedicated field) wins over the host-driver detection.

Environment

  • GPU Operator Version: v26.7.1 (I read v26.7.2 and main and see the same logic, not tested live)
  • DRA driver: v0.5.1, DCGM Exporter 4.6.1-4.8.4-distroless
  • OS: Talos Linux v1.14.5 with the NVIDIA proprietary driver extensions
  • Container Runtime Version: containerd as shipped with Talos v1.14.5
  • Kubernetes Distro and Version: upstream Kubernetes 1.37.1
  • GPU: RTX A1000

Related: #2731

主要言語
Go
スター
2.9k
フォーク
587
平均マージ
2日 2時間
マージ済み PR（30日）
62

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/gpu-operator のほかの issue

NVIDIA/gpu-operator の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。