[Issue]: fused RMSNorm and LayerNorm omit the residual input gradient
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- backend, machine-learning
Research direction
Start with the RMSNorm autograd class in aiter/ops/triton/normalization/rmsnorm.py and the LayerNorm autograd class in aiter/ops/triton/normalization/norm.py, especially the linked backward and saved-tensor logic. Reproduce the issue with the provided script in Triton's interpreter mode and compare both residual gradients with the PyTorch references. Done means both gradients are finite and match the references, including when only the residual requires a gradient.
Written by the indexing model from the issue text.
Description
Describe the bug
The Triton rmsnorm2d_fwd_with_add and layernorm2d_fwd_with_add calls compute normalization of input + residual_in. Their backward methods return no gradient for residual_in, even when it requires a gradient. The script requests that gradient directly and receives None from both calls. The equivalent PyTorch expressions produce finite, nonzero gradients.
The RMSNorm autograd class and LayerNorm autograd class return None in the residual-input position. Their decision to save tensors also checks the input and parameters but omits the residual input. If only the residual requires a gradient, backward therefore lacks the saved tensors it needs. Both classes share the same host-side autograd error.
The linked lines are unchanged on main as of 2026-11-17.
Reproduce
import torch
import torch.nn.functional as F
from aiter.ops.triton.normalization.rmsnorm import rmsnorm2d_fwd_with_add
from aiter.ops.triton.normalization.norm import layernorm2d_fwd_with_add
dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.set_default_device(dev)
torch.manual_seed(0)
checks = []
for name in ('rms', 'layer'):
x = torch.randn(4, 16, requires_grad=True)
residual = torch.randn(4, 16, requires_grad=True)
w, b, dy = torch.randn(16), torch.randn(16), torch.randn(4, 16)
out, residual_out = torch.empty_like(x), torch.empty_like(x)
if name == 'rms':
y = rmsnorm2d_fwd_with_add(out, x, residual, residual_out, w, 1e-5)
s = x + residual
ref = s * torch.rsqrt(s.square().mean(-1, keepdim=True) + 1e-5) * w
else:
y = layernorm2d_fwd_with_add(out, x, residual, residual_out, w, b, 1e-5)
ref = F.layer_norm(x + residual, (16,), w, b, 1e-5)
observed = torch.autograd.grad(y, residual, dy, allow_unused=True)[0]
expected = torch.autograd.grad(ref, residual, dy)[0]
print(name, 'residual gradient:', None if observed is None else observed.flatten()[:3].tolist())
print(name, 'expected:', expected.flatten()[:3].tolist())
checks.append(observed is not None and torch.allclose(observed, expected, atol=1e-4, rtol=1e-4))
assert all(checks), 'Fused normalization drops the residual gradient.'
Output:
rms residual gradient: None
rms expected: [-0.5168098211288452, 0.30581140518188477, -0.21629022061824799]
layer residual gradient: None
layer expected: [-0.19496040046215057, 0.47885698080062866, 0.7730136513710022]
AssertionError: Fused normalization drops the residual gradient.
Expected behavior
For y = norm(x + residual), the residual gradient equals the gradient for x. Both must match the unfused PyTorch expression.
Suggested fix
Include res_in.requires_grad when deciding to save tensors. Return the computed input gradient for both x and residual_in.
Versions / environment
AITER ecd79bcb5d8a (2026-11-17).
torch 2.14.1, triton 3.8.0.
Target configuration: gfx942.
Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
The root cause is host-side autograd configuration and return logic. It is the same on gfx942/gfx950.
- Dominant language
- Python
- Stars
- 616
- Forks
- 709
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 321
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ROCm/aiter
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
Maintainers usually reply within 1 day
-
[Issue]: test_mla_decode_pagesize64 fails instead of skipping on MI450 A0 (B0-only ASM)Possibly taken @feifei14119 claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ROCm/aiter#7779 · 1 assignee ·
Maintainers usually reply within 1 day
Similar issues
-
integration: media_player
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
home-assistant/core#193344 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
daniel-ospina/tortoise#13607 ·
Maintainers usually reply within 1 day
-
documentation priority: medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day