Two 4K@240 Hz DSC displays use all 4 heads on RTX 5080; a third display is rejected (works on Windows)

Start by inspecting the NVIDIA DRM/KMS atomic commit path and display head allocation for DSC modes; the report identifies /proc/driver/nvidia-modeset/heads and GNOME/mutter’s failed atomic commit as evidence. Compare head allocation for two 4K@240 Hz DSC displays with and without the third display, then determine why the commit returns EINVAL and whether the driver can report the constraint.

NVIDIA Open GPU Kernel Modules Version

615.78.08 (open kernel modules)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Arch Linux, kernel 7.2.11-arch1-1

Kernel Release

7.2.11-arch1-1

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 5080 (GB203, 10de:2c02), PNY

Describe the bug

Disclaimer: This ticket was Co-Authored with Claude Opus 5.5 to help me investigate the various pieces. I don't have enough knowledge to debug all those aspects by myself.

With two LG UltraGear+ 4K monitors at 240 Hz over DisplayPort (DSC), the driver appears to use all four display heads. Any additional display then fails to light up, even at 60 Hz, and the compositor's atomic commit fails with EINVAL.

On Windows, the same GPU, monitors, cables and ports run all four displays at full refresh at the same time:

• 2× LG UltraGear+ 3840x2160 @ 240 Hz (DP)
• MSI MAG 273Q X24 2560x1440 @ 240 Hz (DP)
• LG TV 3840x2160 @ 120 Hz (HDMI)

The 4K@240 mode uses a 2332.66 MHz pixel clock (from the EDID DisplayID detailed timing). On Linux, each such display appears to be driven with two heads (2Head1OR), while the Windows driver evidently doesn't need to.

Displays

┌────────────────────┬────────────────────────────────┬───────────────────────┐
│ nvidia-modeset dpy │            Monitor             │      Target mode      │
├────────────────────┼────────────────────────────────┼───────────────────────┤
│ DP-0               │ MSI MAG 273Q X24               │ 2560x1440 @ 240 Hz    │
├────────────────────┼────────────────────────────────┼───────────────────────┤
│ DP-2               │ LG ULTRAGEAR+ (2025 unit)      │ 3840x2160 @ 240.08 Hz │
├────────────────────┼────────────────────────────────┼───────────────────────┤
│ DP-4               │ LG ULTRAGEAR+ (2024 unit)      │ 3840x2160 @ 240.08 Hz │
├────────────────────┼────────────────────────────────┼───────────────────────┤
│ HDMI-0             │ LG TV (unplugged during tests) │ 3840x2160 @ 120 Hz    │
└────────────────────┴────────────────────────────────┴───────────────────────┘

Results

┌────────────────────────────────────┬────────────────────────────────────────────┐
│           Configuration            │                   Result                   │
├────────────────────────────────────┼────────────────────────────────────────────┤
│ LG 240 + LG 240 + MSI 240 + TV     │ ✗ second LG rejected, falls back to 144 Hz │
├────────────────────────────────────┼────────────────────────────────────────────┤
│ LG 240 + LG 144 + MSI 240          │ ✓                                          │
├────────────────────────────────────┼────────────────────────────────────────────┤
│ LG 240 + LG 240 (MSI disabled)     │ ✓ both at 240 Hz                           │
├────────────────────────────────────┼────────────────────────────────────────────┤
│ LG 240 + LG 240 + MSI @ 60 Hz      │ ✗ MSI cannot be enabled                    │
├────────────────────────────────────┼────────────────────────────────────────────┤
│ All four at max refresh on Windows │ ✓                                          │
└────────────────────────────────────┴────────────────────────────────────────────┘

Swapping which LG is active shows that both monitors, cables and ports work at 4K@240 individually.

Evidence

/proc/driver/nvidia-modeset/heads with both LGs at 240 Hz, right after enabling the MSI failed:
head 0 : inactive
head 1 : LG Electronics LG ULTRAGEAR+ (DP-4) 3840 x 2160 @ 240.0844 Hz 30bpp 4:4:4 AdaptiveSync
head 2 : LG Electronics LG ULTRAGEAR+ (DP-2) 3840 x 2160 @ 240.0844 Hz 30bpp 4:4:4 AdaptiveSync
head 3 : inactive
Two heads are reported inactive, yet a third display can't be driven even at 60 Hz. My guess is that heads 0 and 3 are the hidden second heads of the LG displays.

Compositor (GNOME/mutter, Wayland) log when the commit is rejected:
KMS: Posting primary plane composite update for CRTC 1072 (/dev/dri/card1) to DP-3
KMS: [atomic] KMS update failed: drmModeAtomicCommit: Invalid argument
Page flip failed: drmModeAtomicCommit: Invalid argument
The kernel log has no NVRM or nvidia-modeset message explaining the rejection.

Driver configuration: nvidia_drm.modeset=Y, fbdev=Y. No custom registry keys.

To Reproduce

1. Connect two 3840x2160@240 Hz DSC displays and one more display via DP.
2. In a Wayland session, set both 4K displays to 240 Hz.
3. Try to enable the third display at any refresh rate. The atomic commit fails with EINVAL.

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

Expected behavior

The same configuration as on Windows works: both 4K displays at 240 Hz plus additional displays. If two heads per 4K@240 display really is a hardware requirement, the driver should explain why it's rejecting the mode instead of failing silently with EINVAL.

The Windows driver used in the working setup is: GeForce 617.42 WHQL