Under Proton (RADV/ACO), three Model M kernels go over the 192-VGPR bound

Open
#121 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
15/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
cpp

Research direction

The issue asks for a maintainer decision, not a code change: restructure the kernels, report the SPIR-V to Mesa ACO, or leave it. Start with clean/dx12/vgpr_limits.json and clean/dx12/README.md, then the kernels under clean/dx12/kernels built by build_dx12.py. Reproducing the VGPR counts needs RADV_DEBUG=shaderstats under Wine with native_direct_test. Done means a chosen path, agreed with the maintainer, and no code change to the kernels has been made yet.

Written by the indexing model from the issue text.

Description

area: linux area: runtime performance

What

Under Proton, Mesa's ACO compiler needs more VGPRs than clean/dx12/vgpr_limits.json allows for three of Model M's heaviest kernels. Per vgpr_limits.json's own note, they then fit one 256-lane workgroup per CU instead of two:

Kernel VGPRs under ACO Bound Waves/SIMD GPU time per frame
swin_block_c64_nc 240 192 6 0.30 ms
swin_block_c96_pm_rev_nc 256 192 5 0.26 ms
bneck/swin_bneck6_mbo_nc 256 192 5 0.43 ms
swin_block_c96_add_xe_xo_nc 192 192 8 0.21 ms
d1p12/swin_block_c64_add_dec0_nc 192 192 8 0.43 ms

That's about 1.0 ms of Model M's roughly 2.3 ms on the direct route, at 1706×960 → 2560×1440. The repository's notes put a lost workgroup per CU at +38 µs (swin_block_c96_add_xe_* at 215 VGPRs) to +103 µs (dec0 at 201/215 VGPRs) per frame at 1440p Quality. So this is probably a few tenths of a millisecond on Linux. I haven't measured that directly: I can't force ACO to 192 VGPRs to compare.

It isn't loop unrolling. That was my first guess, because of the "unroll loops despite [loop]" note in clean/dx12/README.md, so I checked:

  • swin_block_c64_nc's DXIL has no loops at all: DXC already unrolled it, so AMD's Windows compiler gets the same straight-line code and fits it in ≤192 VGPRs (the build's RGA check).
  • swin_block_c96_pm_rev_nc and the bottleneck keep their loops under ACO (2 and 8 back-edges).

So it's ACO's register allocation on these large kernels. Before scheduling it's already about 198 VGPRs for c64, and 194 and 230 for the other two.

How it was measured (synthetic data only)

  • RX 9070 XT, Mesa RADV 27.0.0-devel, GE-Proton 11-11's Wine, vkd3d-proton 5b1089c8 with dxil-spirv #362 (dxil-spirv 114e664e plus the PR).
  • Kernels built from main's clean/dx12/kernels (unchanged since a8500cb) with build_dx12.py.
  • native_direct_test (direct route, check mode, 3 frames) run under Wine with RADV_DEBUG=shaderstats (VGPRs, waves, LDS) and RADV_DEBUG=shaders (ACO's ISA).
  • Kernels matched by their WMMA count: each SPIR-V cooperative-matrix multiply-add becomes two v_wmma in ACO's ISA (736, 560 and 1312 for the three), and by LDS size where that could be read.
  • GPU times are from WG_D3D12_PROFILE in the same harness. Profiling adds overhead, so treat them as rough.

What could be done (your call):

  • Restructure these kernels so they need fewer registers on any compiler. That would change the Windows DXIL too.
  • Report it to Mesa (ACO) with these kernels' SPIR-V as a reproducer. They're WrongGPU's own MIT-licensed code, so the SPIR-V can be shared. I can put that together if you'd rather it went that way.
  • Leave it. On Linux this is a few tenths of a millisecond, not a correctness problem.

Related: #29, #10.

Dominant language
C++
Stars
12
Forks
2
Avg merge
3h 10m
Merged PRs (30d)
171

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from phantoslayer21/wronggpu

All issues in phantoslayer21/wronggpu

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.