[Issue]: fused RMSNorm and LayerNorm omit the residual input gradient

Open Beginner friendly
#8,138 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Start with the RMSNorm autograd class in aiter/ops/triton/normalization/rmsnorm.py and the LayerNorm autograd class in aiter/ops/triton/normalization/norm.py, especially the linked backward and saved-tensor logic. Reproduce the issue with the provided script in Triton's interpreter mode and compare both residual gradients with the PyTorch references. Done means both gradients are finite and match the references, including when only the residual requires a gradient.

Written by the indexing model from the issue text.

Description

Describe the bug

The Triton rmsnorm2d_fwd_with_add and layernorm2d_fwd_with_add calls compute normalization of input + residual_in. Their backward methods return no gradient for residual_in, even when it requires a gradient. The script requests that gradient directly and receives None from both calls. The equivalent PyTorch expressions produce finite, nonzero gradients.

The RMSNorm autograd class and LayerNorm autograd class return None in the residual-input position. Their decision to save tensors also checks the input and parameters but omits the residual input. If only the residual requires a gradient, backward therefore lacks the saved tensors it needs. Both classes share the same host-side autograd error.

The linked lines are unchanged on main as of 2026-11-17.

Reproduce
import torch
import torch.nn.functional as F
from aiter.ops.triton.normalization.rmsnorm import rmsnorm2d_fwd_with_add
from aiter.ops.triton.normalization.norm import layernorm2d_fwd_with_add

dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.set_default_device(dev)
torch.manual_seed(0)
checks = []
for name in ('rms', 'layer'):
    x = torch.randn(4, 16, requires_grad=True)
    residual = torch.randn(4, 16, requires_grad=True)
    w, b, dy = torch.randn(16), torch.randn(16), torch.randn(4, 16)
    out, residual_out = torch.empty_like(x), torch.empty_like(x)
    if name == 'rms':
        y = rmsnorm2d_fwd_with_add(out, x, residual, residual_out, w, 1e-5)
        s = x + residual
        ref = s * torch.rsqrt(s.square().mean(-1, keepdim=True) + 1e-5) * w
    else:
        y = layernorm2d_fwd_with_add(out, x, residual, residual_out, w, b, 1e-5)
        ref = F.layer_norm(x + residual, (16,), w, b, 1e-5)
    observed = torch.autograd.grad(y, residual, dy, allow_unused=True)[0]
    expected = torch.autograd.grad(ref, residual, dy)[0]
    print(name, 'residual gradient:', None if observed is None else observed.flatten()[:3].tolist())
    print(name, 'expected:', expected.flatten()[:3].tolist())
    checks.append(observed is not None and torch.allclose(observed, expected, atol=1e-4, rtol=1e-4))
assert all(checks), 'Fused normalization drops the residual gradient.'

Output:

rms residual gradient: None
rms expected: [-0.5168098211288452, 0.30581140518188477, -0.21629022061824799]
layer residual gradient: None
layer expected: [-0.19496040046215057, 0.47885698080062866, 0.7730136513710022]
AssertionError: Fused normalization drops the residual gradient.
Expected behavior

For y = norm(x + residual), the residual gradient equals the gradient for x. Both must match the unfused PyTorch expression.

Suggested fix

Include res_in.requires_grad when deciding to save tensors. Return the computed input gradient for both x and residual_in.

Versions / environment

AITER ecd79bcb5d8a (2026-11-17).
torch 2.14.1, triton 3.8.0.
Target configuration: gfx942.
Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
The root cause is host-side autograd configuration and return logic. It is the same on gfx942/gfx950.

Dominant language
Python
Stars
616
Forks
709
Avg merge
3d 10h
Merged PRs (30d)
321

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/aiter

All issues in ROCm/aiter

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.