GPF / use-after-free in postEvent<->osNotifyEvent<->_gpuEngineEventNotificationListNotify under concurrent GPU decode/encode (RTX 5060 Ti, 580.178.04)

Abierto
  • ¿Algo no está bien?
#1,418 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c, linux
Área
embedded-iot, operating-systems

Línea de trabajo

Start by reviewing the reported interrupt trace through postEvent, osNotifyEvent, and _gpuEngineEventNotificationListNotify, then compare it with the supplied vmcore backtraces and matching driver sources. Try the described concurrent NVDEC/NVENC workload if suitable hardware is available; done means identifying and fixing the use-after-free or providing a verified explanation and mitigation. The report says a minimal standalone reproduction is not yet available.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

NVIDIA Open GPU Kernel Modules Version

580.178.04 (Ubuntu package 580.178.04-0ubuntu0.24.04.1, matches upstream tag 580.178.04 exactly)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
  • I confirm that this does not happen with the proprietary driver (of the same version).

Important caveat on the above checkbox: this GPU cannot run the proprietary driver at all — not "doesn't crash," genuinely cannot initialize. Confirmed directly in the kernel log after installing nvidia-driver-580 (proprietary, same 580.178.04 version): RmInitAdapter failed! (0x22:0x56:897) repeating continuously, with NVRM: ...installed in this system requires use of the NVIDIA open kernel modules. nvidia-smi reported "No devices were found" the entire time. So this bug is necessarily open-driver-specific by hardware requirement, not by choice — we could not test the proprietary driver even if we wanted to.

Operating System and Version

Ubuntu 24.04.4 LTS

Kernel Release

Linux qc-rig-1 6.8.0-139-generic #139-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 03:52:05 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

(stock Ubuntu 24.04 HWE kernel package, not self-built)

Please confirm you are running a stable release kernel
  • I am running on a stable kernel release.
Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5060 Ti (UUID: GPU-ff1a37be-b83d-af87-b7ab-7f7454468ead)

Describe the bug

Recurring general protection fault / use-after-free inside the GPU event-notification interrupt path. Crash signature (identical across all 6 occurrences):

RIP: postEvent+77 [nvidia]
RAX: 0x6b6b6b6b6b6b6b6b   <- SLUB freed-memory poison pattern

Call trace (GPU interrupt -> ISR -> notification dispatch -> fault):
asm_exc_general_protection
  [exception RIP: postEvent+77]
osNotifyEvent [nvidia]
_gpuEngineEventNotificationListNotify [nvidia]
kceServiceNotificationInterrupt_IMPL [nvidia]
intrServiceNotificationRecords_IMPL [nvidia]
_intrServiceNonStallLeaf_TU102 [nvidia]
intrServiceNonStall_TU102 [nvidia]
osInterruptPending.constprop.0 [nvidia]
osIsr [nvidia]
rm_isr [nvidia]
nvidia_isr [nvidia]
nvidia_isr_msix [nvidia]
__handle_irq_event_percpu
handle_irq_event
handle_edge_irq
__common_interrupt
common_interrupt
asm_common_interrupt

Occurred 6 times over 2 days (2026-10-03 through 2026-10-05), always during sustained concurrent GPU decode+encode load (multiple simultaneous ffmpeg NVENC/NVDEC sessions via CUDA). The crashing thread in the most recent occurrence was an active HEVC decode worker. In the hours before the 2026-10-04 22:19 occurrence specifically, dmesg shows recurring NVRM: ...Out of memory [NV_ERR_NO_MEMORY] errors roughly every 20-30 minutes, suggesting sustained VRAM pressure is a contributing trigger, not just raw concurrency.

Mitigation applied after the 2nd occurrence: throttled the application's concurrent GPU-decode pipeline from 4 simultaneous jobs to 2. This reduced the crash rate but did not eliminate it (4 more occurrences since).

Why we don't believe this is a known/already-fixed issue: we found a historically similar-looking report (#716, nvbug 4729070, stated fixed in 570.86.15) in this repo, but source-diffed the relevant code across every driver tag from 570.86.15 through our running 580.178.04 and into 590.48.01 -- the fix code in that exact path is byte-identical across all of them. Also, the RTX 5060 Ti (launched ~April 2026) postdates that issue (filed ~October 2025) by roughly 6 months, so it cannot be the same hardware/driver-era recurrence. As far as we can find, this specific fault under sustained concurrent-decode VRAM pressure on Blackwell is undocumented.

To Reproduce
  • Hardware: RTX 5060 Ti, open kernel driver 580.178.04
  • Run 2 (or more) concurrent ffmpeg processes, each doing hardware decode (NVDEC, -hwaccel cuda) of large (multi-GB, 30+ minute) H.264/HEVC source files while simultaneously encoding a separate lower-resolution proxy output via NVENC (-c:v h264_nvenc)
  • Sustain this concurrent load for an extended period (crashes have occurred anywhere from ~38 minutes to ~22 hours into a run -- no fixed reproduction time)
  • Crash occurs during a GPU interrupt, inside the driver's own event-notification dispatch path -- not obviously tied to any specific ffmpeg command-line flag, just sustained concurrent NVENC+NVDEC+CUDA activity

We do not have a minimal standalone repro outside our actual production pipeline yet. Happy to work with NVIDIA to build one if that would help triage.

Bug Incidence

Sometimes

nvidia-bug-report.log.gz

Pending -- will attach via a follow-up comment on this issue shortly (the file is generated and ready, just needs to be dragged into a comment).

More Info

4 full kernel vmcores (captured via kdump) are available for the 4 most recent occurrences, analyzed via the crash utility against matching linux-image-6.8.0-139-generic-dbgsym. Happy to share specific backtrace details, register state, or additional vmcore excerpts on request. This is a 24/7 unattended video-processing pipeline (not an interactive workstation), so the crashes are costly -- currently working around them with a concurrency throttle, but would very much like a real fix or at least guidance on what else to check.

Lenguaje dominante
C
Estrellas
17.5k
Forks
1.9k
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/open-gpu-kernel-modules

Todos los issues de NVIDIA/open-gpu-kernel-modules

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.