[Issue]: chunked_prefill_paged_decode uses the head count for the default attention scale

Open Beginner friendly
#8,136 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
87/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Read aiter/ops/triton/attention/chunked_pa_prefill.py around the documented query shape and default scale calculation. Compare the default scale with the reported query.shape[2] ** -0.5 behavior, then run the reproduction against the PyTorch reference. Done means omitting sm_scale matches the explicit scale within the stated tolerance.

Written by the indexing model from the issue text.

Description

Describe the bug

The documented default for chunked_prefill_paged_decode is 1/sqrt(head_dim). When sm_scale=None, the call instead uses 1/sqrt(num_q_heads). The script uses eight query heads with head dimension 64. The default result differs from the PyTorch attention reference by about 1.38. Passing sm_scale=0.125 reduces the maximum error below 0.002.

The query contract and default calculation describe query shape as (total_tokens, num_q_heads, head_dim), but calculate the scale from query.shape[1]. That axis is the number of heads. The selected scale is passed into the attention kernels. The reproduction uses a full cache page, so unused cache entries do not affect the comparison.

The linked lines are unchanged on main as of 2026-11-17.

Reproduce
import torch
from aiter.ops.triton.attention.chunked_pa_prefill import chunked_prefill_paged_decode

dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.set_default_device(dev)
torch.manual_seed(0)
h, hk, d, bs = 8, 2, 64, 16
query = torch.randn(1, h, d).half()
keys = torch.randn(bs, hk, d).half()
values = torch.randn(bs, hk, d).half()
k_cache = keys.view(1, bs, hk, d // 8, 8).permute(0, 2, 3, 1, 4).contiguous()
v_cache = values.view(1, bs, hk, d).permute(0, 2, 3, 1).contiguous()
block_table = torch.tensor([[0]], dtype=torch.int32)
starts = torch.tensor([0, 1], dtype=torch.int32)
lengths = torch.tensor([bs], dtype=torch.int32)
one = torch.ones(1)
k = keys.float().repeat_interleave(h // hk, 1)
v = values.float().repeat_interleave(h // hk, 1)
p = (torch.einsum('hd,lhd->hl', query[0].float(), k) / d**0.5).softmax(-1)
ref = torch.einsum('hl,lhd->hd', p, v).unsqueeze(0)
outs = []
for scale in (None, d**-0.5):
    out = torch.zeros_like(query)
    chunked_prefill_paged_decode(
        query, keys[-1:], values[-1:], out, 'auto', k_cache, v_cache,
        block_table, starts, lengths, 1, one, one, sm_scale=scale)
    error = (out.float() - ref).abs().max().item()
    print('sm_scale =', scale, '; max error:', round(error, 6), '; allowed: 0.02')
    outs.append(out.float())
assert torch.allclose(outs[1], ref, atol=0.02, rtol=0.02)
assert torch.allclose(outs[0], ref, atol=0.02, rtol=0.02), 'Default scale uses the head count.'

Output:

sm_scale = None ; max error: 1.379511 ; allowed: 0.02
sm_scale = 0.125 ; max error: 0.001859 ; allowed: 0.02
AssertionError: Default scale uses the head count.
Expected behavior

Omitting sm_scale must give the same result as explicitly passing query.shape[2] ** -0.5. The reference uses PyTorch softmax and matrix products.

Suggested fix

Change the default calculation to sm_scale = query.shape[2] ** -0.5.

Versions / environment

AITER ecd79bcb5d8a (2026-11-17).
torch 2.14.1, triton 3.8.0.
Target configuration: gfx942.
Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
The root cause is host-side shape/config logic. The calculation is the same on gfx942/gfx950.

Dominant language
Python
Stars
616
Forks
709
Avg merge
3d 10h
Merged PRs (30d)
321

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/aiter

All issues in ROCm/aiter

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.