GB202: stale WPR2 after abnormal GSP shutdown can block subsequent stock initialization

オープン
#1,442 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
c, linux

調査の方向性

The report identifies the early kgspIsWpr2Up check as the failure point, but does not name source files or tests. Start by locating that check and tracing GSP initialization and recovery on GB202; determine what state validation and supported recovery behavior are safe. Done means a guarded recovery path or documented procedure addresses stale WPR2 state and has appropriate coverage.

索引モデルが issue の本文から書いたものです。

説明

Summary

This is a separate but related recovery problem to #1080.

On an RTX 5090 / GB202 A1, I observed that after an abnormal GSP-RM shutdown or a failed GSP initialization attempt, later stock-driver loads can stop at:

unexpected WPR2 already up, cannot proceed with booting GSP
RmInitAdapter: Cannot initialize GSP firmware RM
rm_init_adapter failed

I am intentionally omitting security-research and trigger details from this public issue. Those details are being provided privately to NVIDIA PSIRT. This issue is limited to the driver/firmware initialization and recovery behavior.

Environment

GPU: RTX 5090 / GB202 A1
Open GPU Kernel Modules: 610.57.04 and 615.71.09
OS: Linux

Observed behavior

In the later stale-state condition:

  • unloading and reloading the driver did not restore GSP-RM initialization;
  • FLR and secondary-bus reset were not reliably sufficient;
  • in a later dirty-state sequence, the WPR2-up condition was observed again after a complete power cycle;
  • the condition remained after six previously changed platform settings were written back to their reference values and read back after a power cycle;
  • the failure occurs at the stock driver's early WPR2 state check, before normal GSP-RM initialization completes.

Private diagnostic testing indicated that the hardware was not necessarily unusable while this stale WPR2 state was present: the normal signed firmware bootstrap could still complete, followed by stable nvidia-smi, CUDA initialization, VRAM allocation, and ordinary compute/display operation. This suggests that at least some stale WPR2 states may be recoverable after validation rather than always being treated as fatal.

Recovery gap

The public driver-facing sequence is:

  1. GSP exits abnormally or initialization fails.
  2. WPR2 remains active or is reconstructed on a later boot.
  3. The next stock initialization sees WPR2 already active and refuses to proceed.
  4. Available reset methods do not reliably return the device to a stock-driver-accepted state.
  5. No documented GB202 teardown, stale-state validation, or service recovery operation is available.

Expected behavior

After an abnormal GSP shutdown or failed initialization, one of the following should be available:

  • firmware-assisted WPR2 teardown;
  • validation and safe reuse of an existing WPR2 state;
  • a guarded stock-driver recovery path;
  • a documented service-level recovery procedure.

Request

Could NVIDIA please:

  1. clarify the intended GB202 recovery path when kgspIsWpr2Up is true before GSP-RM boot;
  2. add a guarded stale-WPR2 validation/teardown path where safe;
  3. document a supported recovery or repair procedure for already affected devices;
  4. advise what logs are useful for diagnosing this condition without exposing security-sensitive reproduction details publicly?

If more information is needed, please reply in this issue with the requested non-sensitive diagnostics. Security-sensitive details can be handled through the associated private PSIRT reports.

主要言語
C
スター
17.5k
フォーク
1.9k
PR マージ指標
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/open-gpu-kernel-modules のほかの issue

NVIDIA/open-gpu-kernel-modules の issue をすべて見る

似ている issue

C の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。