Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: driver-validation hides why a pre-installed driver failed and loops without a reason

Abierto
  • ¿Algo no está bien?
#3,085 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@fjvbn2003 ya está trabajando en esto.

Desde el 16/10/2026.

  • #3086 de @fjvbn2003 — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go, kubernetes
Área
devops

Línea de trabajo

Start from the driver-validation init container's pre-installed driver detection, where No pre-installed driver detected on the host is logged, then follow the fall-through into the retry loop that logs failed to validate the driver, retrying after 5 seconds. Done means the detection log names that nvidia-smi was found but failed, and the retry warning includes the underlying error. Pull request #3086 is already open against this issue, so check it before starting.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

With driver.enabled=false, if nvidia-smi exists on the host but fails (for example Failed to initialize NVML: Driver/library version mismatch after an unattended upgrade of the driver packages without a reboot), the driver-validation init container of nvidia-operator-validator does three things. It logs No pre-installed driver detected on the host: exit status 18. It then falls through to the driver-container path. After that it logs failed to validate the driver, retrying after 5 seconds\n every 5 seconds without the underlying error. The pod stays in Init, and nothing in the validator log says that the host driver is installed but broken.

To Reproduce

  1. Use a node with a pre-installed driver and driver.enabled=false.
  2. Make the host nvidia-smi fail. One way is to upgrade the driver user-space packages without reloading the kernel module. A fake /host/usr/bin/nvidia-smi that prints the mismatch message and exits non-zero gives the same logs.
  3. Read the driver-validation container logs of a fresh nvidia-operator-validator pod on that node.

Expected behavior

The log says that nvidia-smi was found on the host and failed, and the retry warning includes the error it is retrying on.

Environment

  • GPU Operator v26.3.3 (also current main), pre-installed driver on Ubuntu, Kubernetes v1.34, containerd.

Related: #1276, #1687 (users stuck on the same retry line without a reason). I have a small patch ready.

This issue was drafted with AI assistance and reviewed by me.

Lenguaje dominante
Go
Estrellas
2.9k
Forks
576
Merge medio
1 d 9 h
PR fusionados (30 d)
78

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/gpu-operator

Todos los issues de NVIDIA/gpu-operator

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.