LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradients

Open Beginner friendly
#1,647 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

@nazek52 is already working on this.

Since Nov 17, 2026.

  • #1648 by @nazek52 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Start with src/liger_kernel/transformers/poly_norm.py and src/liger_kernel/ops/poly_norm.py, then read test/transformers/test_poly_norm.py and check how it covers the default in_place behavior. Reproduce the residual-gradient and caller-gradient cases from the issue, then verify the expected gradients match the PyTorch reference and the supplied gradient remains unchanged.

Written by the indexing model from the issue text.

Description

Describe the bug

transformers/poly_norm.py#L20 documents in_place as "(default: False)", but #L24 (and ops/poly_norm.py#L356) use True. With in_place=True, backward stores dX into grad_output (ops/poly_norm.py#L296-L297).

So with default arguments:

  • in a residual connection y = norm(x) + x, autograd passes the same gradient tensor to both branches, and the residual branch receives dX instead of the upstream gradient;
  • y.backward(g) overwrites the caller's g.
Reproduce
import torch
from liger_kernel.transformers import LigerPolyNorm

dev = "cuda" if torch.cuda.is_available() else "cpu"

def ref(x, w, b, eps=1e-6):                           # PolyNorm, as in test/transformers/test_poly_norm.py
    n = lambda u: u * torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps)
    return w[0] * n(x ** 3) + w[1] * n(x ** 2) + w[2] * n(x) + b

torch.manual_seed(0)
x = torch.randn(4, 64, device=dev)
g = torch.randn(4, 64, device=dev)
m = LigerPolyNorm().to(dev)                           # default arguments
print("in_place default:", m.in_place, "(docstring says: default False)")
w, b = m.weight.detach().double(), m.bias.detach().double()

x1 = x.clone().requires_grad_()
(m(x1) + x1).backward(g.clone())                      # residual connection
x2 = x.double().requires_grad_()
(ref(x2, w, b) + x2).backward(g.double())
print("y = norm(x) + x: max |x.grad - fp64 reference| =", (x1.grad.double() - x2.grad).abs().max().item())

x3, g3 = x.clone().requires_grad_(), g.clone()
m(x3).backward(g3)
print("y.backward(g): the caller's g changed by", (g3 - g).abs().max().item())

Output:

in_place default: True (docstring says: default False)
y = norm(x) + x: max |x.grad - fp64 reference| = 2.8915757511071902
y.backward(g): the caller's g changed by 2.891575813293457
Expected behavior

x.grad equal to the fp64 PyTorch PolyNorm reference (as in test/transformers/test_poly_norm.py), and g unchanged.

Suggested fix

Use in_place=False as the default (as documented), or detect a shared gradient.

Versions / environment
  • Liger-Kernel 83746f4 (the lines cited above are unchanged on main as of 2026-11-17).
  • torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
  • The root cause is in host-side Python / integer index arithmetic, so it is the same on CUDA. The script runs on cuda when it is available.
Dominant language
Python
Stars
6.7k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Liger-Kernel

All issues in linkedin/Liger-Kernel

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.