GA106 (RTX 3060 12GB, 10de:2504): deterministic GSP fault ~18s after module init on every boot (Xid 119 -> Xid 154), while GB206 cards are healthy in the same boot

Start by reviewing the supplied four-boot kernel log and the GSP timeout and Falcon dump details in the issue; no source files or tests are identified. The report asks maintainers to interpret a firmware fault and advise on further diagnostics, so there is no defined code change or testable completion criterion yet.

NVIDIA Open GPU Kernel Modules Version

610.57.04 — unmodified official release build (build stamp dvs-builder@U22-I3-AF05-32-1, 2026-07-29). Also reproduced with 595.71.05 (unmodified official release build, dvs-builder@U22-I3-G08-03-1, 2026-04-24).

We also tested a self-built 610.57.04 carrying a local P2P patch: it fails identically, and the stock 610.57.04 build fails in exactly the same way, so the local build is not a factor. (In the same boot, three unmodified GB206 cards on this host run on that same self-built module with zero errors.)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

I have not verified this and do not want to tick a box I cannot back up. This host also carries four RTX 5060 Ti (Blackwell), and the proprietary flavour cannot initialise them at all — the kernel log says NVRM: installed in this system requires use of the NVIDIA open kernel modules. followed by RmInitAdapter failed! (0x22:0x56:1050) on all four cards — so the proprietary flavour cannot currently be installed on this machine. All the GA106-only configurations below were tested with open modules only. If you want the proprietary module tested against the GA106, say so and I will physically reconfigure the box (that means pulling the four Blackwell cards) and report back.

Operating System and Version

Ubuntu 26.04.1 LTS

Kernel Release

Linux <host> 7.0.0-34-generic #34-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 2 14:29:37 UTC 2026 x86_64 GNU/Linux

Stock Ubuntu distribution kernel, no local patches. Kernel packages are pinned on this machine.

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GeForce RTX 3060 (UUID: GPU-acc1db81-358d-9650-5b72-8e3858f1dcdc)   <- failing card
       PCI ID 10de:2504 (GA106-302 LHR, 12 GB), board: MSI VENTUS 2X OC LHR

The same machine also hosts 4x 10de:2d04 (GB206 / RTX 5060 Ti 16 GB), which are healthy under the same driver.

Platform: AMD EPYC 7452 (single socket), HUANANZHI H12D-8D v2.0 (BIOS 2.2, 2026-01-06), 128 GB ECC. Each GPU sits on its own CPU root port; there is no PCIe switch / PLX on this board. Kernel command line: ... ro quiet splash amd_iommu=on iommu=pt consoleblank=0 fbcon=nodefer. /proc/driver/nvidia/params: EnableGpuFirmware: 0, EnableResizableBar: 1, DynamicPowerManagement: 3.

Describe the bug

On this host, an RTX 3060 12 GB (GA106-302 LHR, 10de:2504) fails deterministically 8-20 s after the open kernel module finishes loading. It reproduces with the GA106 as the only NVIDIA device in the system, in two different slots, and with two different stock driver branches:

NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  610.57.04  Release Build  (dvs-builder@U22-I3-AF05-32-1)  Wed Jul 29 02:45:17 UTC 2026
NVRM: _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 207235046 heartbeat 207229047 heartbeatWithOffsetMs 207229047 diff 5999 timeout 5200
NVRM: nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:2893
NVRM: _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 76 sequence 2209!
NVRM: _kgspClassifyGspTimeout: ================ GSP core unresponsive ================
NVRM: GPU at PCI:0000:01:00: GPU-acc1db81-358d-9650-5b72-8e3858f1dcdc
NVRM: Xid (PCI:0000:01:00): 119, Timeout after 18s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 2208 (0x20800a70 0x0).
NVRM: _kgspDumpRmState: Failed to get RM state, timeout
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)

After that the device is unusable until reboot (endless NV_ERR_RESET_REQUIRED, kgmmuInvalidateTlb_GM107: TLB invalidation failed ..., nvAssertFailedNoLog: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET)).

Observations that may be useful:

1. The GSP does boot and does run. The RPC history shows roughly 2200 RPCs completing before the stall. The last completed ones are GSP_RM_CONTROL 0x20809064 / 0x20809030 (data1 0x208), and the one that times out is GSP_RM_CONTROL 0x20800a70. So this is not a "GSP failed to start / GFW_BOOT failed" case.

2. The RISC-V (Falcon) core dump printed on timeout is identical across both driver branches, e.g.:

   NVRM: kflcnCoreDumpNondestructive_IMPL: PRI: riscvPc          : 93003a60
   NVRM: kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrStat : 00000000
   NVRM: kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrInfo : badf5620
   NVRM: kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrAddr : 2000000000848048
   NVRM: kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqstat    : 00009000
   

   riscvPrivErrInfo/riscvPrivErrAddr being populated looks like a privileged exception inside the GSP-RM RISC-V firmware rather than a host/GSP handshake problem, but you will read that far better than I can. That is the main reason I am filing this.

3. Only the GA106 is affected. In a boot with the GA106 plus three GB206 (RTX 5060 Ti) on the same module, the three GB206 initialised and stayed healthy — zero Xid lines for them — while the GA106 died with the same Xid 119 within 12 s. So this does not look like a platform-wide or driver-wide problem on this box.

4. The same physical card is fine in Windows 11 on this same machine (nvidia-smi: 12288 MiB, no Xid events; GPU-Z reports MSI VENTUS 2X OC LHR, GA106-302, 12 GB GDDR6, 192-bit).

To Reproduce

1. Install the GA106 (10de:2504, RTX 3060 12 GB) in the machine. It reproduces with the card as the only NVIDIA device present; also reproduced in a second slot (01:00.0 -> c4:00.0, different root port).
2. Boot Ubuntu 26.04.1 LTS with the nvidia open kernel modules 610.57.04 (or 595.71.05) on the stock 7.0.0-34-generic kernel. Nothing beyond the kernel command line above is customised; the module is loaded automatically at boot.
3. Wait ~20 s. No user-space client has to touch the card first — the fault happens on the driver's own initialisation path (in our logs the first CUDA client appears after the Xid).

Result: GSP core unresponsive -> Xid 119 -> Xid 154 -> card unusable.

Bug Incidence

Always

nvidia-bug-report.log.gz

Not attached, and I want to be straight about why: the card has since been physically removed from this server (it is a production box and it went back to running four RTX 5060 Ti for LLM inference), so I cannot run nvidia-bug-report.sh against the failing state right now.

What I have instead, immediately available, is the complete kernel log of four consecutive failing boots, trimmed and sanitised: https://gist.github.com/sean640112/523e33eab4ce969e520b37dcfe30b2d9 — it covers

• GA106 only, 595.71.05, Xid 119 after 18 s
• GA106 only, 610.57.04, Xid 119 after 18 s
• GA106 only, second slot, self-built 610.57.04, Xid 119 after 8 s
• GA106 + 3x GB206, self-built 610.57.04, GA106 dies after 12 s, GB206 healthy

I can put the GA106 back and attach a proper nvidia-bug-report.log.gz from a clean repro (stock 610.57.04 module, matching user space). It is roughly a 10 minute maintenance window on our side. Please just say the word and I will reinstall and attach it; I did not want to hold the report back for days waiting for that window. If your process requires the file before triage, I am happy to have this closed and refile it with the attachment.

More Info

What we already ruled out:

Hypothesis	Test	Result
Bad PCIe slot / link	same fault in 01:00.0 and in c4:00.0 (two different root ports)	Excluded
Driver branch regression	595.71.05 and 610.57.04, both stock official builds	Both fail identically
Our local fork build	stock official build fails the same way	Excluded
GSP firmware blob damaged / mismatched	/lib/firmware/nvidia/610.57.04/gsp_ga10x.bin (84310168 bytes, sha256 c0156954f3e048d56011524e0c2ae2881bb6db8173b53f9b2f4eb94197f02999) is the blob used on this system for both GA10x and GB20x, and it drives three GB206 cards healthily in the same boot	Excluded
GSP never starting (GFW_BOOT)	GSP boots, ~2200 RPCs complete, heartbeats flow and then stop	Excluded
User-space / API mismatch	the fault precedes any CUDA client on the failing boots	Excluded
Runtime power management / GC6 exit	headless server, never suspends; fault occurs ~18 s after module init	Excluded
Hardware fault	same card runs normally under Windows 11 on this machine	Excluded

Not yet tested, and easy for us to test on request: this card with the proprietary flavour (see the checkbox section above), and this card with Resizable BAR disabled in BIOS.

Related issues we found while searching, which look different from ours — #1048 (sporadic Xid 119, RTX 3060 Laptop) and #1294 (GSP GFW_BOOT failure, RTX 3070 Laptop). Ours is deterministic at init on a desktop GA106-302 LHR, and the GSP starts and then faults.

Questions:

1. Is a deterministic GA106 GSP fault like this known internally, or should we treat it as new? If you consider this a duplicate of #1048 or another thread, please close it and point us there — we will move this data into that thread instead.
2. Does the RISC-V/Falcon dump above (riscvPrivErrInfo: badf5620, non-zero riscvPrivErrAddr) point at something specific in GSP-RM, and is there anything more we should capture next time the card is installed (registers, trace buffer, debug builds, additional module parameters)?
3. Is there any branch, firmware blob or module parameter you would like us to try when we put the card back? We can also test the proprietary flavour and BIOS Resizable BAR off in the same maintenance window.

Extra context: we can reproduce this on demand within minutes whenever the card is installed, and we are happy to run any profiling or diagnostics you want.