COpenGLRenderer aborts via RASSERT when the compositor never sends dmabuf feedback (drmFD = -1)

Start in Backend.cpp:109 and follow how IBackend::create() handles the earlier backend failures, then inspect the invalid-fd path in OpenGL.cpp:406-421. Ensure a DRM fd of -1 fails cleanly without reaching the RASSERT, and verify that IBackend::create() returns nullptr instead of aborting.

What happens

IBackend::create() passes g_waylandPlatform->m_drmState.fd straight into
COpenGLRenderer without checking it. If the compositor never delivers linux-dmabuf
feedback, that fd is still -1, and the renderer aborts instead of failing the backend
cleanly.

Backend.cpp:109:

g_openGL = makeShared<COpenGLRenderer>(g_waylandPlatform->m_drmState.fd);

OpenGL.cpp:406-421: eglDeviceFromDRMFD(-1) returns EGL_NO_DEVICE_EXT, so it falls
to the gbm path, openRenderNode(-1) cannot derive a render node, and:

m_gbmFD = CFileDescriptor{openRenderNode(drmFD)};
if (!m_gbmFD.isValid())
    RASSERT(false, "Couldn't open a gbm fd");     // <-- SIGABRT

Note the two paths immediately above this (m_aqBackend->start() failing, and
m_waylandPlatform->attempt() failing) both return nullptr gracefully. Only the
renderer aborts.

Evidence

hyprland-dialog 0.2.2 (shown as Hyprland's "Application Not Responding" watchdog),
core dump read back with gdb + debuginfod:

#3 Hyprtoolkit::COpenGLRenderer::COpenGLRenderer   hyprtoolkit-0.5.4/src/renderer/gl/OpenGL.cpp:421
#5 Hyprtoolkit::IBackend::create                   hyprtoolkit-0.5.4/src/core/Backend.cpp:109
#6 main                                            hyprland-guiutils-0.2.2/utils/dialog/src/main.cpp:108

Platform state read out of the core at the moment of the abort:

m_drmState      = {fd = -1, nodeName = ""}
m_dmabufFormats = std::vector of length 0
dmabuf          = <bound>        dmabufFeedback = <bound>
initialized     = true           dmabufFailed   = false
m_outputs       = length 2

So the client connected fine, bound zwp_linux_dmabuf_v1, created its feedback object
and saw both outputs — the compositor simply never sent main_device or the format
table. dmabufFailed is still false, i.e. nothing in the platform layer noticed.

Why the compositor was silent

The compositor (Hyprland 0.56.2) had exhausted its file descriptors — EMFILE,
confirmed by <pipe(pipefd) failed with24> in its own crash report from the same
moment. Sending dmabuf feedback requires fds it could not allocate. Reported separately
to hyprwm/Hyprland.

That is an unusual state, but it is exactly the state in which a dialog saying
"an application is not responding" is most likely to be launched — so this abort path
is reachable precisely when the dialog is most needed.

Expected

Treat an invalid DRM fd like the other failures in IBackend::create(): log and return
nullptr, letting the caller exit non-zero rather than dumping core. A dialog that
cannot render should exit quietly, not leave a coredump per invocation.

Related

• #126 (POLLHUP on the wayland fd aborts instead of exiting) — same family: RASSERT
  used for a condition that is a normal shutdown, not a fault. Two sibling
  hyprland-dialog processes from the same second as the crash above died inside
  CBackend::enterLoop(), which I believe is #126 rather than this bug.
• #124 is a different hyprland-dialog ANR crash (SIGSEGV, icon theme recursion),
  not this one.

System

hyprtoolkit        0.5.4-4     hyprland-guiutils  0.2.2-2
hyprgraphics       0.5.1-4     hyprutils          0.14.1-1
Hyprland           0.56.2-1
Kernel             7.1.8-arch1-3 (Arch), Omarchy 4.0.1-1
GPU                NVIDIA GeForce RTX 3080, nvidia-open-dkms 610.57.04-1, mesa 1:26.2.1-1

---

Filed by Claude Opus 5 via Claude Code.