LigerFusedLinearCrossEntropyLoss: weight and bias get no gradient when the input does not require grad

Open
#1,644 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

@shotsan is already working on this.

Since Nov 18, 2026.

  • #1654 by @shotsan — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Start with src/liger_kernel/ops/fused_linear_cross_entropy.py, especially the gradient handling around the cited lines 89 and 114–123. Check the existing tests for LigerFusedLinearCrossEntropyLoss and compare gradients with PyTorch when hidden states do not require gradients but weight and bias do. Done means weight.grad and bias.grad match the reference for reduction="mean" and "sum", while the existing "none" behavior still works.

Written by the indexing model from the issue text.

Description

Describe the bug

If the hidden states do not require grad (for example a frozen backbone, and only the lm_head is trained), the loss is correct, but weight.grad and bias.grad stay None. The head is silently not trained. This happens for reduction="mean" and "sum" (the "none" path is correct).

Cause: ops/fused_linear_cross_entropy.py#L89 derives gradient computation from the input only, and #L114-L123 allocates grad_weight / grad_bias only inside if input_requires_grad:; otherwise both are None.

Reproduce
import torch
import torch.nn.functional as F
from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss

dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)
h = torch.randn(8, 16, device=dev)                      # hidden states from a frozen backbone (no grad)
w = torch.randn(32, 16, device=dev, requires_grad=True)  # trainable lm_head
b = torch.randn(32, device=dev, requires_grad=True)
t = torch.randint(0, 32, (8,), device=dev)

loss = LigerFusedLinearCrossEntropyLoss()(w, h, t, b)
loss.backward()

w2, b2 = w.detach().clone().requires_grad_(), b.detach().clone().requires_grad_()
ref = F.cross_entropy(h @ w2.t() + b2, t)
ref.backward()

print("loss:", loss.item(), "torch:", ref.item())
print("weight.grad:", None if w.grad is None else w.grad.abs().max().item(), "| torch max |grad|:", w2.grad.abs().max().item())
print("bias.grad:  ", None if b.grad is None else b.grad.abs().max().item(), "| torch max |grad|:", b2.grad.abs().max().item())

Output:

loss: 7.655911445617676 torch: 7.655911445617676
weight.grad: None | torch max |grad|: 0.42627277970314026
bias.grad:   None | torch max |grad|: 0.22382061183452606
Expected behavior

weight.grad and bias.grad equal to those of F.cross_entropy(h @ w.T + b, t).

Suggested fix

Allocate and compute grad_weight when the weight requires grad and grad_bias when the bias requires grad, independently of the input.

Versions / environment
  • Liger-Kernel 83746f4 (the lines cited above are unchanged on main as of 2026-11-17).
  • torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
  • The root cause is in host-side Python / integer index arithmetic, so it is the same on CUDA. The script runs on cuda when it is available.
Dominant language
Python
Stars
6.7k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Liger-Kernel

All issues in linkedin/Liger-Kernel

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.