GPF / use-after-free in postEvent<->osNotifyEvent<->_gpuEngineEventNotificationListNotify under concurrent GPU decode/encode (RTX 5060 Ti, 580.178.04)

Aperta
  • Qualcosa non va?
#1,418 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, linux
Ambito
embedded-iot, operating-systems

Direzione di ricerca

Start by reviewing the reported interrupt trace through postEvent, osNotifyEvent, and _gpuEngineEventNotificationListNotify, then compare it with the supplied vmcore backtraces and matching driver sources. Try the described concurrent NVDEC/NVENC workload if suitable hardware is available; done means identifying and fixing the use-after-free or providing a verified explanation and mitigation. The report says a minimal standalone reproduction is not yet available.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

NVIDIA Open GPU Kernel Modules Version

580.178.04 (Ubuntu package 580.178.04-0ubuntu0.24.04.1, matches upstream tag 580.178.04 exactly)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
  • I confirm that this does not happen with the proprietary driver (of the same version).

Important caveat on the above checkbox: this GPU cannot run the proprietary driver at all — not "doesn't crash," genuinely cannot initialize. Confirmed directly in the kernel log after installing nvidia-driver-580 (proprietary, same 580.178.04 version): RmInitAdapter failed! (0x22:0x56:897) repeating continuously, with NVRM: ...installed in this system requires use of the NVIDIA open kernel modules. nvidia-smi reported "No devices were found" the entire time. So this bug is necessarily open-driver-specific by hardware requirement, not by choice — we could not test the proprietary driver even if we wanted to.

Operating System and Version

Ubuntu 24.04.4 LTS

Kernel Release

Linux qc-rig-1 6.8.0-139-generic #139-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 03:52:05 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

(stock Ubuntu 24.04 HWE kernel package, not self-built)

Please confirm you are running a stable release kernel
  • I am running on a stable kernel release.
Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5060 Ti (UUID: GPU-ff1a37be-b83d-af87-b7ab-7f7454468ead)

Describe the bug

Recurring general protection fault / use-after-free inside the GPU event-notification interrupt path. Crash signature (identical across all 6 occurrences):

RIP: postEvent+77 [nvidia]
RAX: 0x6b6b6b6b6b6b6b6b   <- SLUB freed-memory poison pattern

Call trace (GPU interrupt -> ISR -> notification dispatch -> fault):
asm_exc_general_protection
  [exception RIP: postEvent+77]
osNotifyEvent [nvidia]
_gpuEngineEventNotificationListNotify [nvidia]
kceServiceNotificationInterrupt_IMPL [nvidia]
intrServiceNotificationRecords_IMPL [nvidia]
_intrServiceNonStallLeaf_TU102 [nvidia]
intrServiceNonStall_TU102 [nvidia]
osInterruptPending.constprop.0 [nvidia]
osIsr [nvidia]
rm_isr [nvidia]
nvidia_isr [nvidia]
nvidia_isr_msix [nvidia]
__handle_irq_event_percpu
handle_irq_event
handle_edge_irq
__common_interrupt
common_interrupt
asm_common_interrupt

Occurred 6 times over 2 days (2026-10-03 through 2026-10-05), always during sustained concurrent GPU decode+encode load (multiple simultaneous ffmpeg NVENC/NVDEC sessions via CUDA). The crashing thread in the most recent occurrence was an active HEVC decode worker. In the hours before the 2026-10-04 22:19 occurrence specifically, dmesg shows recurring NVRM: ...Out of memory [NV_ERR_NO_MEMORY] errors roughly every 20-30 minutes, suggesting sustained VRAM pressure is a contributing trigger, not just raw concurrency.

Mitigation applied after the 2nd occurrence: throttled the application's concurrent GPU-decode pipeline from 4 simultaneous jobs to 2. This reduced the crash rate but did not eliminate it (4 more occurrences since).

Why we don't believe this is a known/already-fixed issue: we found a historically similar-looking report (#716, nvbug 4729070, stated fixed in 570.86.15) in this repo, but source-diffed the relevant code across every driver tag from 570.86.15 through our running 580.178.04 and into 590.48.01 -- the fix code in that exact path is byte-identical across all of them. Also, the RTX 5060 Ti (launched ~April 2026) postdates that issue (filed ~October 2025) by roughly 6 months, so it cannot be the same hardware/driver-era recurrence. As far as we can find, this specific fault under sustained concurrent-decode VRAM pressure on Blackwell is undocumented.

To Reproduce
  • Hardware: RTX 5060 Ti, open kernel driver 580.178.04
  • Run 2 (or more) concurrent ffmpeg processes, each doing hardware decode (NVDEC, -hwaccel cuda) of large (multi-GB, 30+ minute) H.264/HEVC source files while simultaneously encoding a separate lower-resolution proxy output via NVENC (-c:v h264_nvenc)
  • Sustain this concurrent load for an extended period (crashes have occurred anywhere from ~38 minutes to ~22 hours into a run -- no fixed reproduction time)
  • Crash occurs during a GPU interrupt, inside the driver's own event-notification dispatch path -- not obviously tied to any specific ffmpeg command-line flag, just sustained concurrent NVENC+NVDEC+CUDA activity

We do not have a minimal standalone repro outside our actual production pipeline yet. Happy to work with NVIDIA to build one if that would help triage.

Bug Incidence

Sometimes

nvidia-bug-report.log.gz

Pending -- will attach via a follow-up comment on this issue shortly (the file is generated and ready, just needs to be dragged into a comment).

More Info

4 full kernel vmcores (captured via kdump) are available for the 4 most recent occurrences, analyzed via the crash utility against matching linux-image-6.8.0-139-generic-dbgsym. Happy to share specific backtrace details, register state, or additional vmcore excerpts on request. This is a 24/7 unattended video-processing pipeline (not an interactive workstation), so the crashes are costly -- currently working around them with a concurrency throttle, but would very much like a real fix or at least guidance on what else to check.

Lingua principale
C
Stelle
17.5k
Fork
1.9k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/open-gpu-kernel-modules

Tutte le issue di NVIDIA/open-gpu-kernel-modules

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.