LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradients
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- backend, machine-learning
Research direction
Start with src/liger_kernel/transformers/poly_norm.py and src/liger_kernel/ops/poly_norm.py, then read test/transformers/test_poly_norm.py and check how it covers the default in_place behavior. Reproduce the residual-gradient and caller-gradient cases from the issue, then verify the expected gradients match the PyTorch reference and the supplied gradient remains unchanged.
Written by the indexing model from the issue text.
Description
Describe the bug
transformers/poly_norm.py#L20 documents in_place as "(default: False)", but #L24 (and ops/poly_norm.py#L356) use True. With in_place=True, backward stores dX into grad_output (ops/poly_norm.py#L296-L297).
So with default arguments:
- in a residual connection
y = norm(x) + x, autograd passes the same gradient tensor to both branches, and the residual branch receives dX instead of the upstream gradient; y.backward(g)overwrites the caller'sg.
Reproduce
import torch
from liger_kernel.transformers import LigerPolyNorm
dev = "cuda" if torch.cuda.is_available() else "cpu"
def ref(x, w, b, eps=1e-6): # PolyNorm, as in test/transformers/test_poly_norm.py
n = lambda u: u * torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps)
return w[0] * n(x ** 3) + w[1] * n(x ** 2) + w[2] * n(x) + b
torch.manual_seed(0)
x = torch.randn(4, 64, device=dev)
g = torch.randn(4, 64, device=dev)
m = LigerPolyNorm().to(dev) # default arguments
print("in_place default:", m.in_place, "(docstring says: default False)")
w, b = m.weight.detach().double(), m.bias.detach().double()
x1 = x.clone().requires_grad_()
(m(x1) + x1).backward(g.clone()) # residual connection
x2 = x.double().requires_grad_()
(ref(x2, w, b) + x2).backward(g.double())
print("y = norm(x) + x: max |x.grad - fp64 reference| =", (x1.grad.double() - x2.grad).abs().max().item())
x3, g3 = x.clone().requires_grad_(), g.clone()
m(x3).backward(g3)
print("y.backward(g): the caller's g changed by", (g3 - g).abs().max().item())
Output:
in_place default: True (docstring says: default False)
y = norm(x) + x: max |x.grad - fp64 reference| = 2.8915757511071902
y.backward(g): the caller's g changed by 2.891575813293457
Expected behavior
x.grad equal to the fp64 PyTorch PolyNorm reference (as in test/transformers/test_poly_norm.py), and g unchanged.
Suggested fix
Use in_place=False as the default (as documented), or detect a shared gradient.
Versions / environment
- Liger-Kernel
83746f4(the lines cited above are unchanged onmainas of 2026-11-17). - torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (
TRITON_INTERPRET=1). - The root cause is in host-side Python / integer index arithmetic, so it is the same on CUDA. The script runs on
cudawhen it is available.
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
linkedin/Liger-Kernel#1659 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day