[Bug]: 09_gemm_one_shot_all_reduce aborts on power-of-two CU counts (gemm_sms auto-detect equals total_sms)
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- distributed-systems, performance
Research direction
The bug is in examples/09_gemm_one_shot_all_reduce/benchmark.py lines 92-96 and 142-144. Start by reading the auto-detection logic for gemm_sms and total_sms. Run the failing command to reproduce the error. The fix involves adjusting the power-of-two calculation to ensure gemm_sms is strictly less than total_sms. Check the comment about reserving SMs for communication. After the fix, test with a GPU that has a power-of-two CU count to confirm the example runs.
Written by the indexing model from the issue text.
Description
Describe the bug
examples/09_gemm_one_shot_all_reduce/benchmark.py aborts before doing any
work on a GPU whose CU (SM) count is already a power of two. The
auto-detection computes a value and then the guard rejects that same value.
Where
benchmark.py:92-96 and benchmark.py:142-144:
cu_count = torch.cuda.get_device_properties(rank).multi_processor_count
if args["total_sms"] is None:
args["total_sms"] = cu_count
if args["gemm_sms"] is None:
# For all_reduce: use next smaller power of 2, rest for communication
args["gemm_sms"] = 2 ** int(math.log2(cu_count)) if cu_count > 0 else 1
...
if args["gemm_sms"] >= args["total_sms"]:
print(f"Invalid number of GEMM SMs. {args['gemm_sms']} >= {args['total_sms']}")
exit(1)
When cu_count is a power of two, 2 ** floor(log2(cu_count)) == cu_count, so
gemm_sms == total_sms and the guard fires. The comment says "next smaller
power of 2", but for an exact power of two the expression returns the value
itself, not the next smaller one — hence no SMs left for communication,
which is presumably what the guard is actually trying to prevent.
To Reproduce
Any GPU with a power-of-two CU count. Measured on an RX 9070 XT (gfx1201),
where multi_processor_count == 32:
$ python benchmark.py --validate --datatype fp16 -m 512 -n 512 -k 512 --num_ranks 1
Invalid number of GEMM SMs. 32 >= 32
No --gemm_sms/--total_sms are passed; both are auto-detected. The same
happens at 64 CUs by inspection (e.g. Vega/gfx900 here reports 64), and at
16/128/256 on other parts. It does not occur on a non-power-of-two count
(e.g. 6 CUs on an iGPU computes gemm_sms=4 < total_sms=6, and proceeds).
Expected behavior
The example runs with its auto-detected defaults on a power-of-two-CU GPU.
Suggested fix
Either halve, or take the next smaller power of two strictly less than the
count:
args["gemm_sms"] = 2 ** (int(math.log2(cu_count)) - 1) if cu_count > 1 else 1
(or max(1, cu_count // 2), which is simpler and also reserves a meaningful
share for communication).
I have not submitted this as a patch because the right split between GEMM and
communication SMs is a performance decision the maintainers own, not something
I should pick unilaterally — but happy to open a PR with whichever form you
prefer.
Environment
- OS: Arch Linux / CachyOS, kernel 7.2.6-1-cachyos
- ROCm 7.14 (torch reports
hip: 7.14.60850), torch 2.12.0+rocm7.14, triton 3.7.1 - iris: main @ 5b46b8e
- GPU: AMD Radeon RX 9070 XT,
gfx1201,multi_processor_count = 32
Related
Found while writing the pytest coverage for this example (#63). Working around
it with --gemm_sms 16 --total_sms 32 the example then hits a second
pre-existing issue — the default BLK_M/BLK_N of 256x256 requires 131072
bytes of shared memory against a 65536-byte limit:
triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 131072, Hardware limit: 65536
128x128 validates cleanly (Final C validation passed.). I can file that
separately too if useful — mentioning it here only so the two are not
confused, since fixing the SM guard alone still leaves the default block size
unusable on this target.
- Dominant language
- Python
- Stars
- 202
- Forks
- 52
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 12
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ROCm/iris
-
benchmarks bug examples iris test
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 3 days
-
iris
Difficulty 5/5 Over a week Newbie friendliness 15/100
Maintainers usually reply within 3 days
-
benchmarks gluon iris
Difficulty 5/5 Over a week Newbie friendliness 30/100
Maintainers usually reply within 3 days
-
bug iris test
Difficulty 4/5 3-5 days Newbie friendliness 40/100
Maintainers usually reply within 3 days
-
[Feature]: Formalize allocator-agnostic symmetric tensor address translationMay be free again @artulab claimed this 49 days ago, and no pull request is open. Openiris
ROCm/iris#546 · 2 comments · 1 assignee ·
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
uniswap-python/uniswap-python#487 ·
-
[Bug]: agent_final_response still returns the abandoned branch's answer after navigate moves HEAD backPossibly taken @enyst claimed this today. Openbug priority:low ready-for-dev
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
OpenHands/software-agent-sdk#5822 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
area: server enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
scs0209/laya-guard#52 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 2 days