#[Bug] Cross entropy, fused CE+TVD and GRPO's log-softmax return NaN when a row's first block of logits is all -inf
I maintainer di solito rispondono entro 3 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 55/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Ambito
- machine-learning
Direzione di ricerca
Read the online softmax update in src/liger_kernel/ops/cross_entropy.py, src/liger_kernel/ops/fused_ce_tvd.py, and src/liger_kernel/ops/grpo_loss.py, along with the fix pattern in issue #1525. Reproduce the leading all--inf block case for cross entropy, fused CE+TVD, and GRPO, including the sequence-level GRPO path mentioned in the issue. Done means these paths return finite results matching PyTorch, with finite gradients where applicable.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
🐛 Describe the bug
liger_cross_entropy_kernel, _fused_ce_tvd_forward_kernel, _selective_log_softmax_kernel and _grpo_loss_fwd_kernel compute each row's log-sum-exp with an online softmax over blocks of logits, starting from max = -inf, sumexp = 0. If every logit of the row's first block is -inf, the running max is still -inf after it:
m_new = tl.maximum(m, block_max) # -inf
d = d * tl.exp2((m - m_new) * LOG2_E) + tl.sum(tl.exp2((X_block - m_new) * LOG2_E))
m = m_new
Both m - m_new and X_block - m_new are -inf - (-inf) = NaN, so the running sum becomes NaN and stays NaN through the remaining blocks. The row's log-sum-exp is NaN, and so is everything computed from it:
liger_cross_entropy_kernel: the loss and every entry of the gradient._fused_ce_tvd_forward_kernel: the CE, the TVD and the saved log-sum-exp. This happens for the student logits and, by the same code, for the teacher's (then the TVD is NaN)._selective_log_softmax_kernel(fused_selective_log_softmax) and_grpo_loss_fwd_kernel: the row's log-probability, and from it the GRPO loss.
The rows are valid: they have finite logits after the first block, and the label points at one of them. torch.nn.functional.cross_entropy and torch.softmax give finite results for them.
This needs a vocabulary larger than one block (32768 logits for cross entropy, 16384 for fused CE+TVD, 2048 for the GRPO kernels) whose first block is all masked, e.g. by logits.masked_fill(~allowed, float("-inf")) when only tokens past the first block's ids are allowed. LigerCrossEntropyLoss and LigerFusedLinearCrossEntropyLoss both launch the cross-entropy kernel.
It is the same pattern #1525 fixes in _softmax_multi_block_forward_kernel. On current main (781a8ae):
liger_cross_entropy_kernelL139, L183–L185_fused_ce_tvd_forward_kernelL96–L107_selective_log_softmax_kernelL58–L60_grpo_loss_fwd_kernelL158–L160
Reproduce
Cross entropy:
import torch
from liger_kernel.transformers import LigerCrossEntropyLoss
torch.manual_seed(0)
V = 50257
x = torch.randn(4, V, device="cuda")
x[:, :32768] = float("-inf") # the first block masked
y = torch.randint(32768, V, (4,), device="cuda") # labels among the unmasked logits
print(torch.nn.functional.cross_entropy(x, y, reduction="none"))
xg = x.clone().requires_grad_(True)
loss = LigerCrossEntropyLoss(reduction="none")(xg, y)
loss.sum().backward()
print(loss, xg.grad.isnan().sum().item())
| loss | NaN gradient entries | |
|---|---|---|
torch |
[10.2685, 11.4343, 10.1693, 9.7875] |
0 |
Liger main |
[nan, nan, nan, nan] |
201028 (all) |
| with the fix below | [10.2685, 11.4343, 10.1693, 9.7875] |
0 |
With the fix, the gradient matches torch's to 5e-10.
Fused CE+TVD:
import torch
from liger_kernel.ops.fused_ce_tvd import LigerFusedCETVDFunction
torch.manual_seed(0)
V = 50257
s = torch.randn(4, V, device="cuda")
t = torch.randn(4, V, device="cuda")
s[:, :16384] = float("-inf") # the student's first block masked
y = torch.randint(16384, V, (4,), device="cuda")
ce, tvd = LigerFusedCETVDFunction.apply(s, t, y, -100)
print(ce, tvd)
print(torch.nn.functional.cross_entropy(s, y, reduction="none"),
0.5 * (torch.softmax(s, -1) - torch.softmax(t, -1)).abs().sum(-1))
| CE | TVD | |
|---|---|---|
torch |
[11.0964, 9.6125, 12.2725, 12.3862] |
[0.6105, 0.6117, 0.6126, 0.6141] |
Liger main |
[nan, nan, nan, nan] |
[nan, nan, nan, nan] |
| with the fix below | [11.0964, 9.6125, 12.2725, 12.3862] |
[0.6105, 0.6117, 0.6126, 0.6141] |
With the teacher's first block masked instead, main gives a NaN TVD, and the fix matches torch to 1.2e-7. With no masked logits, both kernels agree with torch before and after the fix.
GRPO's selective log-softmax:
import torch
from liger_kernel.ops.grpo_loss import fused_selective_log_softmax
torch.manual_seed(0)
B, L, N = 2, 3, 4096
logits = torch.randn(B, L + 1, N, device="cuda")
logits[..., :2048] = float("-inf") # the first block (BLOCK_N = 2048) masked
input_ids = torch.randint(2048, N, (B, L), device="cuda")
print(fused_selective_log_softmax(logits, input_ids, temperature=0.9))
print(torch.log_softmax(logits[:, :-1] / 0.9, -1).gather(-1, input_ids[..., None])[..., 0])
| log-probabilities | |
|---|---|
torch |
[[-7.1026, -8.7077, -8.2264], [-9.7784, -6.3727, -11.8797]] |
Liger main |
[[nan, nan, nan], [nan, nan, nan]] |
| with the fix below | [[-7.1026, -8.7077, -8.2264], [-9.7784, -6.3727, -11.8797]] |
Suggested fix
Rescale against 0 while the running max is still -inf, as #1525 does for softmax. At that point the running sum is still 0, so the result is unchanged.
m_new = tl.maximum(m, block_max)
- d = d * tl.exp2((m - m_new) * LOG2_E) + tl.sum(tl.exp2((X_block - m_new) * LOG2_E))
+ # Avoid -inf - (-inf) = NaN while every logit so far is -inf.
+ ref_m = tl.where(m_new == float("-inf"), 0.0, m_new)
+ d = d * tl.exp2((m - ref_m) * LOG2_E) + tl.sum(tl.exp2((X_block - ref_m) * LOG2_E))
m = m_new
_fused_ce_tvd_forward_kernel takes the same change for student_sumexp and teacher_sumexp, and the two GRPO kernels for alpha and l_i. The full patch for the three files, against 781a8ae, is in Liger_online_softmax_masked_leading_block.patch.
Repository follow-up (2026-11-02): the tracked _grpo_loss_fwd_kernel_seq also uses this update.
The local fix.patch now covers that sequence-level path. Its original capture is
reproduced before generating the corrected PTX with the original binding; the verification records
are in the GRPO follow-up.
This follow-up adds no GPU measurements to the reproduction results above.
Reproduce
No response
Versions
RTX 4060 Ti (sm_89), torch 2.9.1+cu128, Triton 3.5.1, Liger main (781a8ae).
- Lingua principale
- Python
- Stelle
- 6.7k
- Fork
- 668
- Merge medio
- 4g 12h
- PR unite (30g)
- 27
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di linkedin/Liger-Kernel
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
linkedin/Liger-Kernel#1658 ·
I maintainer di solito rispondono entro 3 giorni
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsForse già presa @nazek52 l’ha presa 1 giorno fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
linkedin/Liger-Kernel#1647 ·
I maintainer di solito rispondono entro 3 giorni
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Forse già presa @0xDexe l’ha presa 8 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
linkedin/Liger-Kernel#1570 · 3 commenti ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 68/100
linkedin/Liger-Kernel#1661 ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 45/100
linkedin/Liger-Kernel#1659 ·
I maintainer di solito rispondono entro 3 giorni
Tutte le issue di linkedin/Liger-Kernel
Issue simili
-
Internal ErrorApertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
frappe/frappe_docker#2071 ·
I maintainer di solito rispondono entro 1 giorno
-
bug python
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
traceloop/openllmetry#4717 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
The-Pocket/PocketFlow#161 ·
-
[Bug]: Webapp workspace fails to render in Safari — "Can't find variable: requestIdleCallback"Apertacomp/desktop P2 type/bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
NousResearch/hermes-agent#157702 ·
I maintainer di solito rispondono entro 1 giorno
-
scibench_physics / scibench_chemistry: a correct answer scores 0 when the unit has a power of tenForse già presa @BahaaMattar l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
EvolvingLMMs-Lab/lmms-eval#1699 ·
I maintainer di solito rispondono entro 1 giorno