[MLA][gfx942] Native qh64 fp8 persistent decode kernel GPU-faults at page_size=1 (block_size=1)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- Half a day
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
Research direction
Start with aiter/mla.py and run op_tests/op_test_mla_qh64_gfx942_pagesize1.py on a gfx942 MI300X to reproduce the page_size=1 fault. Compare the page_size=1 and page_size=64 dispatch paths; done means the validated native path remains for page_size=64, page_size=1 avoids the faulty kernel, and the regression test passes.
Written by the indexing model from the issue text.
Description
Summary
The native qh64 fp8 persistent MLA-decode kernel added in #3188
(hsa/gfx942/mla/mla_a8w8_qh64_qseqlen1_gqaratio64_v3_ps.co) GPU
memory-access-faults on gfx942 (MI300X) at page_size=1 (block_size=1).
In aiter/mla.py, mla_decode_fwd routes
get_gfx() in ("gfx942","gfx950") and nhead==64 and q.dtype==fp8 and kv.dtype==fp8 and max_seqlen_q==1 to this kernel. #3188 validated it only at page_size=64;
all configs in that PR discussion use page_size=64. Production vLLM MLA serves
with page_size=1, which the kernel never exercised — it does an out-of-bounds
access there and kills the worker on the first decode forward.
Reproduction
Single MI300X (gfx942), stock aiter carrying #3188 (e.g. v0.1.18):
[aiter] LoadKernel: _ZN5aiter39mla_a8w8_qh64_qseqlen1_gqaratio64_v3_psE hsaco:
.../gfx942/mla/mla_a8w8_qh64_qseqlen1_gqaratio64_v3_ps.co
[test] mla_decode_fwd persistent batch=1 kv_len=512 ... finite=True
[test] mla_decode_fwd persistent batch=4 kv_len=1024 ... finite=True
[test] mla_decode_fwd persistent batch=16 kv_len=2048 ...
Memory access fault by GPU node-2 (Agent handle: 0x1011fc10) on address 0x7f5bd0ed9000. Reason: Unknown.
Aborted (exit 134)
Confirmed on 3 independent aiter builds carrying #3188 (e03fa6040+cherry-pick
04427a5d8, ac90d5c89, and origin/main tip) and on multiple MI300X nodes.
The .co blob is byte-identical from #3188 through tip, so later metadata changes
do not avoid it — the OOB is in the kernel binary.
Context
Surfaced on GLM-5.1-FP8 (GlmMoeDsaForCausalLM, MLA + DSA) in WideEP
1P1D / 2P2D disaggregated serving, where per-rank head grouping is
gqa_ratio=64 (data_parallel_size=8, tensor_parallel_size=1).
Note this is the opposite regime from #2821 (missing gqa=8 kernels at TP=8):
here the gqa=64 kernel exists but is broken at page_size=1.
Fix (dispatch-level guard; PR to follow)
Restrict the gfx942 native-qh64 clause to the validated page_size==64; other page
sizes (page_size=1) fall through to the pre-#3188 qh16 fold, which runs without
fault. gfx950 unchanged. A standalone repro
(op_tests/test_mla_qh64_gfx942_pagesize1.py) and the one-line dispatch guard are
ready to submit as a draft PR. The proper long-term fix is the kernel's page_size=1
addressing bug.
Env: ROCm 7.2.3, MI300X (gfx942), AITER v0.1.18, PyTorch 2.11.
Related: #3188, #2821.
- Dominant language
- Python
- Stars
- 565
- Forks
- 585
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 371
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ROCm/aiter
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100