GB202: stale WPR2 after abnormal GSP shutdown can block subsequent stock initialization

Abierto
  • ¿Algo no está bien?
#1,442 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c, linux

Línea de trabajo

The report identifies the early kgspIsWpr2Up check as the failure point, but does not name source files or tests. Start by locating that check and tracing GSP initialization and recovery on GB202; determine what state validation and supported recovery behavior are safe. Done means a guarded recovery path or documented procedure addresses stale WPR2 state and has appropriate coverage.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

This is a separate but related recovery problem to #1080.

On an RTX 5090 / GB202 A1, I observed that after an abnormal GSP-RM shutdown or a failed GSP initialization attempt, later stock-driver loads can stop at:

unexpected WPR2 already up, cannot proceed with booting GSP
RmInitAdapter: Cannot initialize GSP firmware RM
rm_init_adapter failed

I am intentionally omitting security-research and trigger details from this public issue. Those details are being provided privately to NVIDIA PSIRT. This issue is limited to the driver/firmware initialization and recovery behavior.

Environment

GPU: RTX 5090 / GB202 A1
Open GPU Kernel Modules: 610.57.04 and 615.71.09
OS: Linux

Observed behavior

In the later stale-state condition:

  • unloading and reloading the driver did not restore GSP-RM initialization;
  • FLR and secondary-bus reset were not reliably sufficient;
  • in a later dirty-state sequence, the WPR2-up condition was observed again after a complete power cycle;
  • the condition remained after six previously changed platform settings were written back to their reference values and read back after a power cycle;
  • the failure occurs at the stock driver's early WPR2 state check, before normal GSP-RM initialization completes.

Private diagnostic testing indicated that the hardware was not necessarily unusable while this stale WPR2 state was present: the normal signed firmware bootstrap could still complete, followed by stable nvidia-smi, CUDA initialization, VRAM allocation, and ordinary compute/display operation. This suggests that at least some stale WPR2 states may be recoverable after validation rather than always being treated as fatal.

Recovery gap

The public driver-facing sequence is:

  1. GSP exits abnormally or initialization fails.
  2. WPR2 remains active or is reconstructed on a later boot.
  3. The next stock initialization sees WPR2 already active and refuses to proceed.
  4. Available reset methods do not reliably return the device to a stock-driver-accepted state.
  5. No documented GB202 teardown, stale-state validation, or service recovery operation is available.

Expected behavior

After an abnormal GSP shutdown or failed initialization, one of the following should be available:

  • firmware-assisted WPR2 teardown;
  • validation and safe reuse of an existing WPR2 state;
  • a guarded stock-driver recovery path;
  • a documented service-level recovery procedure.

Request

Could NVIDIA please:

  1. clarify the intended GB202 recovery path when kgspIsWpr2Up is true before GSP-RM boot;
  2. add a guarded stale-WPR2 validation/teardown path where safe;
  3. document a supported recovery or repair procedure for already affected devices;
  4. advise what logs are useful for diagnosing this condition without exposing security-sensitive reproduction details publicly?

If more information is needed, please reply in this issue with the requested non-sensitive diagnostics. Security-sensitive details can be handled through the associated private PSIRT reports.

Lenguaje dominante
C
Estrellas
17.5k
Forks
1.9k
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/open-gpu-kernel-modules

Todos los issues de NVIDIA/open-gpu-kernel-modules

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.