LigerFusedLinearCrossEntropyLoss: weight and bias get no gradient when the input does not require grad

Aperta
#1,644 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@shotsan ci sta già lavorando.

Dal 18/11/2026.

  • #1654 di @shotsan — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
68/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch

Direzione di ricerca

Start with src/liger_kernel/ops/fused_linear_cross_entropy.py, especially the gradient handling around the cited lines 89 and 114–123. Check the existing tests for LigerFusedLinearCrossEntropyLoss and compare gradients with PyTorch when hidden states do not require gradients but weight and bias do. Done means weight.grad and bias.grad match the reference for reduction="mean" and "sum", while the existing "none" behavior still works.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

If the hidden states do not require grad (for example a frozen backbone, and only the lm_head is trained), the loss is correct, but weight.grad and bias.grad stay None. The head is silently not trained. This happens for reduction="mean" and "sum" (the "none" path is correct).

Cause: ops/fused_linear_cross_entropy.py#L89 derives gradient computation from the input only, and #L114-L123 allocates grad_weight / grad_bias only inside if input_requires_grad:; otherwise both are None.

Reproduce
import torch
import torch.nn.functional as F
from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss

dev = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)
h = torch.randn(8, 16, device=dev)                      # hidden states from a frozen backbone (no grad)
w = torch.randn(32, 16, device=dev, requires_grad=True)  # trainable lm_head
b = torch.randn(32, device=dev, requires_grad=True)
t = torch.randint(0, 32, (8,), device=dev)

loss = LigerFusedLinearCrossEntropyLoss()(w, h, t, b)
loss.backward()

w2, b2 = w.detach().clone().requires_grad_(), b.detach().clone().requires_grad_()
ref = F.cross_entropy(h @ w2.t() + b2, t)
ref.backward()

print("loss:", loss.item(), "torch:", ref.item())
print("weight.grad:", None if w.grad is None else w.grad.abs().max().item(), "| torch max |grad|:", w2.grad.abs().max().item())
print("bias.grad:  ", None if b.grad is None else b.grad.abs().max().item(), "| torch max |grad|:", b2.grad.abs().max().item())

Output:

loss: 7.655911445617676 torch: 7.655911445617676
weight.grad: None | torch max |grad|: 0.42627277970314026
bias.grad:   None | torch max |grad|: 0.22382061183452606
Expected behavior

weight.grad and bias.grad equal to those of F.cross_entropy(h @ w.T + b, t).

Suggested fix

Allocate and compute grad_weight when the weight requires grad and grad_bias when the bias requires grad, independently of the input.

Versions / environment
  • Liger-Kernel 83746f4 (the lines cited above are unchanged on main as of 2026-11-17).
  • torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
  • The root cause is in host-side Python / integer index arithmetic, so it is the same on CUDA. The script runs on cuda when it is available.
Lingua principale
Python
Stelle
6.7k
Fork
668
Merge medio
4g 12h
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di linkedin/Liger-Kernel

Tutte le issue di linkedin/Liger-Kernel

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.