[MLA][gfx942] Native qh64 fp8 persistent decode kernel GPU-faults at page_size=1 (block_size=1)

Open Beginner friendly
#4,363 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
Half a day
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Start with aiter/mla.py and run op_tests/op_test_mla_qh64_gfx942_pagesize1.py on a gfx942 MI300X to reproduce the page_size=1 fault. Compare the page_size=1 and page_size=64 dispatch paths; done means the validated native path remains for page_size=64, page_size=1 avoids the faulty kernel, and the regression test passes.

Written by the indexing model from the issue text.

Description

Summary

The native qh64 fp8 persistent MLA-decode kernel added in #3188
(hsa/gfx942/mla/mla_a8w8_qh64_qseqlen1_gqaratio64_v3_ps.co) GPU
memory-access-faults on gfx942 (MI300X) at page_size=1 (block_size=1)
.

In aiter/mla.py, mla_decode_fwd routes
get_gfx() in ("gfx942","gfx950") and nhead==64 and q.dtype==fp8 and kv.dtype==fp8 and max_seqlen_q==1 to this kernel. #3188 validated it only at page_size=64;
all configs in that PR discussion use page_size=64. Production vLLM MLA serves
with page_size=1, which the kernel never exercised — it does an out-of-bounds
access there and kills the worker on the first decode forward.

Reproduction

Single MI300X (gfx942), stock aiter carrying #3188 (e.g. v0.1.18):

[aiter] LoadKernel: _ZN5aiter39mla_a8w8_qh64_qseqlen1_gqaratio64_v3_psE hsaco:
        .../gfx942/mla/mla_a8w8_qh64_qseqlen1_gqaratio64_v3_ps.co
[test] mla_decode_fwd persistent batch=1  kv_len=512  ... finite=True
[test] mla_decode_fwd persistent batch=4  kv_len=1024 ... finite=True
[test] mla_decode_fwd persistent batch=16 kv_len=2048 ...
Memory access fault by GPU node-2 (Agent handle: 0x1011fc10) on address 0x7f5bd0ed9000. Reason: Unknown.
Aborted (exit 134)

Confirmed on 3 independent aiter builds carrying #3188 (e03fa6040+cherry-pick
04427a5d8, ac90d5c89, and origin/main tip) and on multiple MI300X nodes.
The .co blob is byte-identical from #3188 through tip, so later metadata changes
do not avoid it — the OOB is in the kernel binary.

Context

Surfaced on GLM-5.1-FP8 (GlmMoeDsaForCausalLM, MLA + DSA) in WideEP
1P1D / 2P2D disaggregated serving, where per-rank head grouping is
gqa_ratio=64 (data_parallel_size=8, tensor_parallel_size=1).

Note this is the opposite regime from #2821 (missing gqa=8 kernels at TP=8):
here the gqa=64 kernel exists but is broken at page_size=1.

Fix (dispatch-level guard; PR to follow)

Restrict the gfx942 native-qh64 clause to the validated page_size==64; other page
sizes (page_size=1) fall through to the pre-#3188 qh16 fold, which runs without
fault. gfx950 unchanged. A standalone repro
(op_tests/test_mla_qh64_gfx942_pagesize1.py) and the one-line dispatch guard are
ready to submit as a draft PR. The proper long-term fix is the kernel's page_size=1
addressing bug.

Env: ROCm 7.2.3, MI300X (gfx942), AITER v0.1.18, PyTorch 2.11.

Related: #3188, #2821.

Dominant language
Python
Stars
565
Forks
585
Avg merge
3d 1h
Merged PRs (30d)
371

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/aiter

All issues in ROCm/aiter

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.