GB202: stale WPR2 after abnormal GSP shutdown can block subsequent stock initialization

未关闭
  • 哪里不对？
#1,442 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
c, linux

调研方向

The report identifies the early kgspIsWpr2Up check as the failure point, but does not name source files or tests. Start by locating that check and tracing GSP initialization and recovery on GB202; determine what state validation and supported recovery behavior are safe. Done means a guarded recovery path or documented procedure addresses stale WPR2 state and has appropriate coverage.

由索引模型根据 Issue 内容生成。

描述

Summary

This is a separate but related recovery problem to #1080.

On an RTX 5090 / GB202 A1, I observed that after an abnormal GSP-RM shutdown or a failed GSP initialization attempt, later stock-driver loads can stop at:

unexpected WPR2 already up, cannot proceed with booting GSP
RmInitAdapter: Cannot initialize GSP firmware RM
rm_init_adapter failed

I am intentionally omitting security-research and trigger details from this public issue. Those details are being provided privately to NVIDIA PSIRT. This issue is limited to the driver/firmware initialization and recovery behavior.

Environment

GPU: RTX 5090 / GB202 A1
Open GPU Kernel Modules: 610.57.04 and 615.71.09
OS: Linux

Observed behavior

In the later stale-state condition:

  • unloading and reloading the driver did not restore GSP-RM initialization;
  • FLR and secondary-bus reset were not reliably sufficient;
  • in a later dirty-state sequence, the WPR2-up condition was observed again after a complete power cycle;
  • the condition remained after six previously changed platform settings were written back to their reference values and read back after a power cycle;
  • the failure occurs at the stock driver's early WPR2 state check, before normal GSP-RM initialization completes.

Private diagnostic testing indicated that the hardware was not necessarily unusable while this stale WPR2 state was present: the normal signed firmware bootstrap could still complete, followed by stable nvidia-smi, CUDA initialization, VRAM allocation, and ordinary compute/display operation. This suggests that at least some stale WPR2 states may be recoverable after validation rather than always being treated as fatal.

Recovery gap

The public driver-facing sequence is:

  1. GSP exits abnormally or initialization fails.
  2. WPR2 remains active or is reconstructed on a later boot.
  3. The next stock initialization sees WPR2 already active and refuses to proceed.
  4. Available reset methods do not reliably return the device to a stock-driver-accepted state.
  5. No documented GB202 teardown, stale-state validation, or service recovery operation is available.

Expected behavior

After an abnormal GSP shutdown or failed initialization, one of the following should be available:

  • firmware-assisted WPR2 teardown;
  • validation and safe reuse of an existing WPR2 state;
  • a guarded stock-driver recovery path;
  • a documented service-level recovery procedure.

Request

Could NVIDIA please:

  1. clarify the intended GB202 recovery path when kgspIsWpr2Up is true before GSP-RM boot;
  2. add a guarded stale-WPR2 validation/teardown path where safe;
  3. document a supported recovery or repair procedure for already affected devices;
  4. advise what logs are useful for diagnosing this condition without exposing security-sensitive reproduction details publicly?

If more information is needed, please reply in this issue with the requested non-sensitive diagnostics. Security-sensitive details can be handled through the associated private PSIRT reports.

主要语言
C
星标
17.5k
派生
1.9k
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/open-gpu-kernel-modules 的其他 Issue

查看 NVIDIA/open-gpu-kernel-modules 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。