Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Issue]: KFD runlist oversubscription at exactly 8 CP queue slots causes sclk pin + preemption hangs on RDNA2 (gfx1030) under 3+ concurrent ROCm processes — root cause + fix (amdgpu.num_kcq)

Open
  • Is something wrong?
#227 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
c, linux
Domain
operating-systems

Research direction

Start by reviewing the kernel 6.8 amdgpu/amdkfd source around pm_calc_rlib_size() and the amdgpu.num_kcq parameter. Reproduce the three-process workload, then inspect sysfs and dmesg for queue usage and oversubscription lines. Done means determining whether the default is intentional and validating any proposed queue-budget change without the reported hangs.

Written by the indexing model from the issue text.

Description

Title

[Issue]: KFD runlist oversubscription at exactly 8 CP queue slots causes sclk pin + preemption hangs on RDNA2 (gfx1030) under 3+ concurrent ROCm processes — root cause + fix (amdgpu.num_kcq)

Environment

  • GPU: AMD Radeon PRO V620 (Navi 21 / gfx1030)
  • OS: Ubuntu 24.04.4 LTS
  • Kernel: 6.8.0-137-generic
  • ROCm: 7.2.4
  • Workload: 2-3 concurrent llama.cpp server processes (HIP backend), each an independent long-lived process holding its own HSA queues — not a single multi-stream process.

Symptom

Under sustained 3-process concurrency on one card:

  • GPU core clock (sclk) pins at maximum continuously, even when all three processes are otherwise idle between requests (~42 W idle-but-pinned vs a true idle floor well under that).
  • Reported GPU busy% reads ~99% even when no process is actively decoding — the busy metric does not reflect real work.
  • Under load, threads intermittently hang stuck in ioctl on /dev/kfd inside libhsa-runtime64, requiring a process kill to recover.

Root cause (traced against kernel 6.8 amdgpu/amdkfd source, verified against sysfs on this box)

Navi 21 exposes 16 HQD compute-queue slots on MEC0. The amdgpu kernel driver reserves amdgpu.num_kcq of those as kernel compute rings — default 8 — leaving exactly 8 CP queue slots for all ROCm userspace processes on the card combined.

Each concurrent HIP process consumes multiple CP queues (in our testing, ~3 per process: 2 HIP-visible + 1 ROCr-internal queue that isn't covered by GPU_MAX_HW_QUEUES). At 3 concurrent processes this reaches or exceeds the 8-queue budget. KFD's runlist-size calculation (pm_calc_rlib_size()) detects the oversubscription and emits a chained runlist instead of a flat one; the MEC hardware scheduler then round-robins the queue sets on a fixed timer quantum indefinitely. This means the CP front-end never truly idles (the sclk pin, and the misleading busy% reading), and the constant evict/preempt cycling this produces exercises gfx10's known-fragile preemption path — which is where we see the hangs.

This budget is considerably tighter than AMD's own Instinct-class oversubscription guidance implies (that guidance describes 24 queues; gfx9 has 8 queues/pipe where RDNA2 has 4).

Fix, measured

  1. Partial mitigation: GPU_MAX_HW_QUEUES=2 per process. Caps the HIP-visible queue count per process (AMD's own workload-tuning guidance already recommends =2 for single-stream llama.cpp-style workloads), no throughput cost measured for a single compute stream. This raises the ceiling from ~daily failures to only failing under full 3-process co-residency, because the ROCr-internal queue isn't covered by this cap (still ~3 queues/process floor).
  2. Full fix: amdgpu.num_kcq=2 kernel module parameter, raising the userspace CP queue budget from 8 to 14. With this + the GPU_MAX_HW_QUEUES=2 cap, 3 fully concurrent processes measured at 9/14 queues used, zero oversubscription lines in dmesg, and — notably — the card reached a genuine 3-process idle state for the first time (sclk 0, ~7 W, 0% busy), confirming the busy%/clock-pin symptom above was entirely the oversubscription artifact, not real GPU load.

Question for maintainers

Is the num_kcq=8 default (reserving half of Navi 21's 16 HQD slots for kernel rings) intended to leave RDNA2 users this little multi-process headroom, or is this closer to a bringup-era default that hasn't been revisited for the "several independent long-lived ROCm processes sharing one consumer/workstation card" use case? Related reports we found describe the same symptom class on RDNA3/RDNA4 (pinned clocks, hangs that don't reproduce on Vulkan) but none isolated this specific mechanism:

  • ROCm/TheRock#5793 (gfx1102, resolved via sched_policy=1 rather than the queue budget)
  • ROCm/ROCm#5706 (gfx1201, unresolved on ROCm, users moved to Vulkan)
  • ROCm/amdgpu#153 (RDNA3, multiple HW queues via MES, 100% GPU usage)

Happy to provide the full sysfs dump, dmesg oversubscription lines, or test additional configurations if useful.

Dominant language
C
Stars
463
Forks
145
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/amdgpu

All issues in ROCm/amdgpu

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.