nvkms-hdmi.c: hblankMin overflows 32 bits and ignores lane count / FRL coding for 3-lane DSC configs

Open
  • Is something wrong?
#1,445 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c
Domain
operating-systems

Research direction

The bug is in pDscInfo->hdmi.hblankMin inside nvHdmiFrlQueryConfigOneColorFormatAndBpc() in src/nvidia-modeset/src/nvkms-hdmi.c. Read how GetHdmiFrlLinkRate() maps the FRL rate to lanes, then check that the numerator is computed in 64 bits and the divisor uses the lane count and 16b/18b coding. Done when the 3L6G example (1697380 kHz, dscHBlankTriBytes 72) yields about 184 instead of 2. No test file is named, so verify with the logged values in the issue.

Written by the indexing model from the issue text.

Description

Summary

nvHdmiFrlQueryConfigOneColorFormatAndBpc() in src/nvidia-modeset/src/nvkms-hdmi.c computes pDscInfo->hdmi.hblankMin with two arithmetic errors:

  1. 32-bit overflow. The numerator is evaluated in 32 bits and wraps for typical high-refresh 4K modes.
  2. Wrong tri-byte rate for 3-lane FRL configs. The divisor assumes 4 lanes and ignores 16b/18b FRL coding.

I found these while debugging a separate HDMI audio problem (filed as #1446). Fixing them did not fix that audio problem, so this is reported as its own bug, with no claim that it explains the audio loss.

Affected code

pDscInfo->hdmi.hblankMin =
    NV_UNSIGNED_DIV_CEIL(((pHwTimings->pixelClock * 1000) *
                              pConfig->dscInfo.dscHBlankTriBytes),
                         (hdmiLinkRate / 6));
1. Overflow

pixelClock is in kHz (NvU32). pixelClock * 1000 * dscHBlankTriBytes is computed in 32 bits.

Example: a 3840x2160 @ 165 Hz mode with a 1697380 kHz pixel clock and dscHBlankTriBytes = 72 gives 1,697,380,000 x 72, which wraps in 32 bits. The result was hblankMin = 2 instead of roughly 122 to 184.

2. Lane count and FRL coding

GetHdmiFrlLinkRate() returns the per-lane rate, and / 6 corresponds to 4 lanes x rate / 24 bits per tri-byte. For the 3-lane rates (3LANES_3GBPS, 3LANES_6GBPS) the divisor should use 3 lanes. FRL 16b/18b character coding is not accounted for either.

For 3L6G the actual tri-byte rate is 6e9 x 3 x (16/18) / 24, about 6.67e8 per second. 72 tri-bytes then need about 108 ns, which is about 184 pixel times at 1697 MHz.

Observed values

Instrumented stock code, 3840x2160 @ 165 Hz, 10 bpc, FRL 3L6G, DSC:

frlcfg pclkKHz=1697380 raster=4400x2338 frlRate=2 (3LANES_6GBPS) dsc=1 hcBlankTri=72 tBlankRatioX1k=30 hblankMin=2
hblank rasterX=4400 hBlank=560 hblankMin=2 tBlankRatioX1k=30 target=132

With the fix below, hblankMin is about 184 for this config. For 4L6G (HCblank = 280 tri-bytes) it comes out at 535.

In this case max(hblankMin, rasterX * tBlankRatio / 1000) currently masks the error (target = 132). After the correction the target is 184. Other modes or sinks may not be masked.

Suggested fix

{
    const NvU64 frlLanes =
        ((pConfig->frlRate == HDMI_FRL_DATA_RATE_3LANES_3GBPS) ||
         (pConfig->frlRate == HDMI_FRL_DATA_RATE_3LANES_6GBPS)) ? 3 : 4;
    /* tri-bytes/s = laneRate * lanes * (16/18) / 24 */
    const NvU64 triBytesPerSec = (hdmiLinkRate * frlLanes * 16) / (18 * 24);
    const NvU64 pixelClockHz = (NvU64)pHwTimings->pixelClock * 1000;

    pDscInfo->hdmi.hblankMin = (NvU32)
        NV_UNSIGNED_DIV_CEIL(pixelClockHz * pConfig->dscInfo.dscHBlankTriBytes,
                             triBytesPerSec);
}

Tested: builds and runs on 615.78.08 with GB203. Video is unaffected, and the logged values match the hand calculation. I could only test the 3L6G and 4L6G configs on one sink.

Environment

  • Driver: 615.78.08 (open kernel modules)
  • GPU: GeForce RTX 5090 Laptop GPU (GB203)
  • Display: Samsung Odyssey Ark over HDMI 2.1 (max FRL 4x10G, DSC 1.2a)
  • Kernel: 7.2.9, Fedora 44 based (Bazzite)
Dominant language
C
Stars
17.5k
Forks
1.9k
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/open-gpu-kernel-modules

All issues in NVIDIA/open-gpu-kernel-modules

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.