LigerFusedLinearCrossEntropyLoss: weight and bias get no gradient when the input does not require grad
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- machine-learning, performance
Research direction
Start with src/liger_kernel/ops/fused_linear_cross_entropy.py, especially the gradient handling around the cited lines 89 and 114–123. Check the existing tests for LigerFusedLinearCrossEntropyLoss and compare gradients with PyTorch when hidden states do not require gradients but weight and bias do. Done means weight.grad and bias.grad match the reference for reduction="mean" and "sum", while the existing "none" behavior still works.
Written by the indexing model from the issue text.
Description
Describe the bug
If the hidden states do not require grad (for example a frozen backbone, and only the lm_head is trained), the loss is correct, but weight.grad and bias.grad stay None. The head is silently not trained. This happens for reduction="mean" and "sum" (the "none" path is correct).
Cause: ops/fused_linear_cross_entropy.py#L89 derives gradient computation from the input only, and #L114-L123 allocates grad_weight / grad_bias only inside if input_requires_grad:; otherwise both are None.
Reproduce
import torch
import torch.nn.functional as F
from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss
dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)
h = torch.randn(8, 16, device=dev) # hidden states from a frozen backbone (no grad)
w = torch.randn(32, 16, device=dev, requires_grad=True) # trainable lm_head
b = torch.randn(32, device=dev, requires_grad=True)
t = torch.randint(0, 32, (8,), device=dev)
loss = LigerFusedLinearCrossEntropyLoss()(w, h, t, b)
loss.backward()
w2, b2 = w.detach().clone().requires_grad_(), b.detach().clone().requires_grad_()
ref = F.cross_entropy(h @ w2.t() + b2, t)
ref.backward()
print("loss:", loss.item(), "torch:", ref.item())
print("weight.grad:", None if w.grad is None else w.grad.abs().max().item(), "| torch max |grad|:", w2.grad.abs().max().item())
print("bias.grad: ", None if b.grad is None else b.grad.abs().max().item(), "| torch max |grad|:", b2.grad.abs().max().item())
Output:
loss: 7.655911445617676 torch: 7.655911445617676
weight.grad: None | torch max |grad|: 0.42627277970314026
bias.grad: None | torch max |grad|: 0.22382061183452606
Expected behavior
weight.grad and bias.grad equal to those of F.cross_entropy(h @ w.T + b, t).
Suggested fix
Allocate and compute grad_weight when the weight requires grad and grad_bias when the bias requires grad, independently of the input.
Versions / environment
- Liger-Kernel
83746f4(the lines cited above are unchanged onmainas of 2026-11-17). - torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (
TRITON_INTERPRET=1). - The root cause is in host-side Python / integer index arithmetic, so it is the same on CUDA. The script runs on
cudawhen it is available.
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day