[Issue]: chunked_prefill_paged_decode uses the head count for the default attention scale
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 87/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- backend, machine-learning
Research direction
Read aiter/ops/triton/attention/chunked_pa_prefill.py around the documented query shape and default scale calculation. Compare the default scale with the reported query.shape[2] ** -0.5 behavior, then run the reproduction against the PyTorch reference. Done means omitting sm_scale matches the explicit scale within the stated tolerance.
Written by the indexing model from the issue text.
Description
Describe the bug
The documented default for chunked_prefill_paged_decode is 1/sqrt(head_dim). When sm_scale=None, the call instead uses 1/sqrt(num_q_heads). The script uses eight query heads with head dimension 64. The default result differs from the PyTorch attention reference by about 1.38. Passing sm_scale=0.125 reduces the maximum error below 0.002.
The query contract and default calculation describe query shape as (total_tokens, num_q_heads, head_dim), but calculate the scale from query.shape[1]. That axis is the number of heads. The selected scale is passed into the attention kernels. The reproduction uses a full cache page, so unused cache entries do not affect the comparison.
The linked lines are unchanged on main as of 2026-11-17.
Reproduce
import torch
from aiter.ops.triton.attention.chunked_pa_prefill import chunked_prefill_paged_decode
dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.set_default_device(dev)
torch.manual_seed(0)
h, hk, d, bs = 8, 2, 64, 16
query = torch.randn(1, h, d).half()
keys = torch.randn(bs, hk, d).half()
values = torch.randn(bs, hk, d).half()
k_cache = keys.view(1, bs, hk, d // 8, 8).permute(0, 2, 3, 1, 4).contiguous()
v_cache = values.view(1, bs, hk, d).permute(0, 2, 3, 1).contiguous()
block_table = torch.tensor([[0]], dtype=torch.int32)
starts = torch.tensor([0, 1], dtype=torch.int32)
lengths = torch.tensor([bs], dtype=torch.int32)
one = torch.ones(1)
k = keys.float().repeat_interleave(h // hk, 1)
v = values.float().repeat_interleave(h // hk, 1)
p = (torch.einsum('hd,lhd->hl', query[0].float(), k) / d**0.5).softmax(-1)
ref = torch.einsum('hl,lhd->hd', p, v).unsqueeze(0)
outs = []
for scale in (None, d**-0.5):
out = torch.zeros_like(query)
chunked_prefill_paged_decode(
query, keys[-1:], values[-1:], out, 'auto', k_cache, v_cache,
block_table, starts, lengths, 1, one, one, sm_scale=scale)
error = (out.float() - ref).abs().max().item()
print('sm_scale =', scale, '; max error:', round(error, 6), '; allowed: 0.02')
outs.append(out.float())
assert torch.allclose(outs[1], ref, atol=0.02, rtol=0.02)
assert torch.allclose(outs[0], ref, atol=0.02, rtol=0.02), 'Default scale uses the head count.'
Output:
sm_scale = None ; max error: 1.379511 ; allowed: 0.02
sm_scale = 0.125 ; max error: 0.001859 ; allowed: 0.02
AssertionError: Default scale uses the head count.
Expected behavior
Omitting sm_scale must give the same result as explicitly passing query.shape[2] ** -0.5. The reference uses PyTorch softmax and matrix products.
Suggested fix
Change the default calculation to sm_scale = query.shape[2] ** -0.5.
Versions / environment
AITER ecd79bcb5d8a (2026-11-17).
torch 2.14.1, triton 3.8.0.
Target configuration: gfx942.
Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
The root cause is host-side shape/config logic. The calculation is the same on gfx942/gfx950.
- Dominant language
- Python
- Stars
- 616
- Forks
- 709
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 321
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ROCm/aiter
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
[Issue]: test_mla_decode_pagesize64 fails instead of skipping on MI450 A0 (B0-only ASM)Possibly taken @feifei14119 claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ROCm/aiter#7779 · 1 assignee ·
Maintainers usually reply within 1 day
Similar issues
-
integration: media_player
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
home-assistant/core#193344 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
daniel-ospina/tortoise#13607 ·
Maintainers usually reply within 1 day
-
documentation priority: medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day