#[Bug] Cross entropy, fused CE+TVD and GRPO's log-softmax return NaN when a row's first block of logits is all -inf

Open
#1,660 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Read the online softmax update in src/liger_kernel/ops/cross_entropy.py, src/liger_kernel/ops/fused_ce_tvd.py, and src/liger_kernel/ops/grpo_loss.py, along with the fix pattern in issue #1525. Reproduce the leading all--inf block case for cross entropy, fused CE+TVD, and GRPO, including the sequence-level GRPO path mentioned in the issue. Done means these paths return finite results matching PyTorch, with finite gradients where applicable.

Written by the indexing model from the issue text.

Description

🐛 Describe the bug

liger_cross_entropy_kernel, _fused_ce_tvd_forward_kernel, _selective_log_softmax_kernel and _grpo_loss_fwd_kernel compute each row's log-sum-exp with an online softmax over blocks of logits, starting from max = -inf, sumexp = 0. If every logit of the row's first block is -inf, the running max is still -inf after it:

        m_new = tl.maximum(m, block_max)   # -inf
        d = d * tl.exp2((m - m_new) * LOG2_E) + tl.sum(tl.exp2((X_block - m_new) * LOG2_E))
        m = m_new

Both m - m_new and X_block - m_new are -inf - (-inf) = NaN, so the running sum becomes NaN and stays NaN through the remaining blocks. The row's log-sum-exp is NaN, and so is everything computed from it:

  • liger_cross_entropy_kernel: the loss and every entry of the gradient.
  • _fused_ce_tvd_forward_kernel: the CE, the TVD and the saved log-sum-exp. This happens for the student logits and, by the same code, for the teacher's (then the TVD is NaN).
  • _selective_log_softmax_kernel (fused_selective_log_softmax) and _grpo_loss_fwd_kernel: the row's log-probability, and from it the GRPO loss.

The rows are valid: they have finite logits after the first block, and the label points at one of them. torch.nn.functional.cross_entropy and torch.softmax give finite results for them.

This needs a vocabulary larger than one block (32768 logits for cross entropy, 16384 for fused CE+TVD, 2048 for the GRPO kernels) whose first block is all masked, e.g. by logits.masked_fill(~allowed, float("-inf")) when only tokens past the first block's ids are allowed. LigerCrossEntropyLoss and LigerFusedLinearCrossEntropyLoss both launch the cross-entropy kernel.

It is the same pattern #1525 fixes in _softmax_multi_block_forward_kernel. On current main (781a8ae):

Reproduce

Cross entropy:

import torch
from liger_kernel.transformers import LigerCrossEntropyLoss

torch.manual_seed(0)
V = 50257
x = torch.randn(4, V, device="cuda")
x[:, :32768] = float("-inf")                     # the first block masked
y = torch.randint(32768, V, (4,), device="cuda")  # labels among the unmasked logits

print(torch.nn.functional.cross_entropy(x, y, reduction="none"))
xg = x.clone().requires_grad_(True)
loss = LigerCrossEntropyLoss(reduction="none")(xg, y)
loss.sum().backward()
print(loss, xg.grad.isnan().sum().item())
loss NaN gradient entries
torch [10.2685, 11.4343, 10.1693, 9.7875] 0
Liger main [nan, nan, nan, nan] 201028 (all)
with the fix below [10.2685, 11.4343, 10.1693, 9.7875] 0

With the fix, the gradient matches torch's to 5e-10.

Fused CE+TVD:

import torch
from liger_kernel.ops.fused_ce_tvd import LigerFusedCETVDFunction

torch.manual_seed(0)
V = 50257
s = torch.randn(4, V, device="cuda")
t = torch.randn(4, V, device="cuda")
s[:, :16384] = float("-inf")                     # the student's first block masked
y = torch.randint(16384, V, (4,), device="cuda")

ce, tvd = LigerFusedCETVDFunction.apply(s, t, y, -100)
print(ce, tvd)
print(torch.nn.functional.cross_entropy(s, y, reduction="none"),
      0.5 * (torch.softmax(s, -1) - torch.softmax(t, -1)).abs().sum(-1))
CE TVD
torch [11.0964, 9.6125, 12.2725, 12.3862] [0.6105, 0.6117, 0.6126, 0.6141]
Liger main [nan, nan, nan, nan] [nan, nan, nan, nan]
with the fix below [11.0964, 9.6125, 12.2725, 12.3862] [0.6105, 0.6117, 0.6126, 0.6141]

With the teacher's first block masked instead, main gives a NaN TVD, and the fix matches torch to 1.2e-7. With no masked logits, both kernels agree with torch before and after the fix.

GRPO's selective log-softmax:

import torch
from liger_kernel.ops.grpo_loss import fused_selective_log_softmax

torch.manual_seed(0)
B, L, N = 2, 3, 4096
logits = torch.randn(B, L + 1, N, device="cuda")
logits[..., :2048] = float("-inf")               # the first block (BLOCK_N = 2048) masked
input_ids = torch.randint(2048, N, (B, L), device="cuda")

print(fused_selective_log_softmax(logits, input_ids, temperature=0.9))
print(torch.log_softmax(logits[:, :-1] / 0.9, -1).gather(-1, input_ids[..., None])[..., 0])
log-probabilities
torch [[-7.1026, -8.7077, -8.2264], [-9.7784, -6.3727, -11.8797]]
Liger main [[nan, nan, nan], [nan, nan, nan]]
with the fix below [[-7.1026, -8.7077, -8.2264], [-9.7784, -6.3727, -11.8797]]
Suggested fix

Rescale against 0 while the running max is still -inf, as #1525 does for softmax. At that point the running sum is still 0, so the result is unchanged.

         m_new = tl.maximum(m, block_max)
-        d = d * tl.exp2((m - m_new) * LOG2_E) + tl.sum(tl.exp2((X_block - m_new) * LOG2_E))
+        # Avoid -inf - (-inf) = NaN while every logit so far is -inf.
+        ref_m = tl.where(m_new == float("-inf"), 0.0, m_new)
+        d = d * tl.exp2((m - ref_m) * LOG2_E) + tl.sum(tl.exp2((X_block - ref_m) * LOG2_E))
         m = m_new

_fused_ce_tvd_forward_kernel takes the same change for student_sumexp and teacher_sumexp, and the two GRPO kernels for alpha and l_i. The full patch for the three files, against 781a8ae, is in Liger_online_softmax_masked_leading_block.patch.

Repository follow-up (2026-11-02): the tracked _grpo_loss_fwd_kernel_seq also uses this update.
The local fix.patch now covers that sequence-level path. Its original capture is
reproduced before generating the corrected PTX with the original binding; the verification records
are in the GRPO follow-up.
This follow-up adds no GPU measurements to the reproduction results above.

Reproduce

No response

Versions

RTX 4060 Ti (sm_89), torch 2.9.1+cu128, Triton 3.5.1, Liger main (781a8ae).

Dominant language
Python
Stars
6.7k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Liger-Kernel

All issues in linkedin/Liger-Kernel

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.