GB202: stale WPR2 after abnormal GSP shutdown can block subsequent stock initialization

Aperta
  • Qualcosa non va?
#1,442 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, linux

Direzione di ricerca

The report identifies the early kgspIsWpr2Up check as the failure point, but does not name source files or tests. Start by locating that check and tracing GSP initialization and recovery on GB202; determine what state validation and supported recovery behavior are safe. Done means a guarded recovery path or documented procedure addresses stale WPR2 state and has appropriate coverage.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

This is a separate but related recovery problem to #1080.

On an RTX 5090 / GB202 A1, I observed that after an abnormal GSP-RM shutdown or a failed GSP initialization attempt, later stock-driver loads can stop at:

unexpected WPR2 already up, cannot proceed with booting GSP
RmInitAdapter: Cannot initialize GSP firmware RM
rm_init_adapter failed

I am intentionally omitting security-research and trigger details from this public issue. Those details are being provided privately to NVIDIA PSIRT. This issue is limited to the driver/firmware initialization and recovery behavior.

Environment

GPU: RTX 5090 / GB202 A1
Open GPU Kernel Modules: 610.57.04 and 615.71.09
OS: Linux

Observed behavior

In the later stale-state condition:

  • unloading and reloading the driver did not restore GSP-RM initialization;
  • FLR and secondary-bus reset were not reliably sufficient;
  • in a later dirty-state sequence, the WPR2-up condition was observed again after a complete power cycle;
  • the condition remained after six previously changed platform settings were written back to their reference values and read back after a power cycle;
  • the failure occurs at the stock driver's early WPR2 state check, before normal GSP-RM initialization completes.

Private diagnostic testing indicated that the hardware was not necessarily unusable while this stale WPR2 state was present: the normal signed firmware bootstrap could still complete, followed by stable nvidia-smi, CUDA initialization, VRAM allocation, and ordinary compute/display operation. This suggests that at least some stale WPR2 states may be recoverable after validation rather than always being treated as fatal.

Recovery gap

The public driver-facing sequence is:

  1. GSP exits abnormally or initialization fails.
  2. WPR2 remains active or is reconstructed on a later boot.
  3. The next stock initialization sees WPR2 already active and refuses to proceed.
  4. Available reset methods do not reliably return the device to a stock-driver-accepted state.
  5. No documented GB202 teardown, stale-state validation, or service recovery operation is available.

Expected behavior

After an abnormal GSP shutdown or failed initialization, one of the following should be available:

  • firmware-assisted WPR2 teardown;
  • validation and safe reuse of an existing WPR2 state;
  • a guarded stock-driver recovery path;
  • a documented service-level recovery procedure.

Request

Could NVIDIA please:

  1. clarify the intended GB202 recovery path when kgspIsWpr2Up is true before GSP-RM boot;
  2. add a guarded stale-WPR2 validation/teardown path where safe;
  3. document a supported recovery or repair procedure for already affected devices;
  4. advise what logs are useful for diagnosing this condition without exposing security-sensitive reproduction details publicly?

If more information is needed, please reply in this issue with the requested non-sensitive diagnostics. Security-sensitive details can be handled through the associated private PSIRT reports.

Lingua principale
C
Stelle
17.5k
Fork
1.9k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/open-gpu-kernel-modules

Tutte le issue di NVIDIA/open-gpu-kernel-modules

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.