triton_grpo_loss: zero gradient with delta when the unclipped branch wins (negative advantage)

Open
#1,643 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

@shotsan is already working on this.

Since Nov 18, 2026.

  • #1655 by @shotsan — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
77/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Read src/liger_kernel/ops/grpo_loss.py around the token and sequence level backward logic at the cited lines, then compare its behavior with torch_grpo_loss in test/transformers/test_grpo_loss.py. Run the provided reproduction in Triton's interpreter mode and check both token and sequence cases. Done means the loss remains correct and the gradient matches the reference for negative advantages when delta is set.

Written by the indexing model from the issue text.

Description

Describe the bug

With delta set (two-sided clipping), the loss is -min(min(r, delta) * A, clip(r, 1-eps_low, 1+eps_high) * A). When A < 0 and delta < r < 1 + eps_high, min selects the second branch. In that branch r is not clipped, so the gradient is -r * A * dlogp, which is not zero. The kernel returns a zero gradient. The loss value is correct.

Cause: ops/grpo_loss.py#L482-L486:

mask = per_token_loss2 >= per_token_loss1
dlogp = -coef_1 * advantage * mask          # gradient only if branch 1 wins
if DELTA != 0.0:
    dlogp = dlogp * (coef_1 <= DELTA)

This assumes that "branch 2 wins" means "coef_2 is clipped". That is true without delta, but with delta, branch 2 can win while coef_2 = coef_1 is inside the clip range. The sequence-level backward has the same logic (#L377-L383).

Reproduce
import math
import torch
from liger_kernel.transformers.grpo_loss import triton_grpo_loss

dev = "cuda" if torch.cuda.is_available() else "cpu"
# 1 sequence, 1 completion token, vocab 2. log p(token) = log(1/2).
# old_logp is chosen so that ratio = p / p_old = 1.4:
#   1.4 > delta = 1.25, and 1.4 is inside [1 - eps_low, 1 + eps_high] = [0.75, 1.5]. Advantage = -1.
logits = torch.zeros(1, 2, 2, device=dev, requires_grad=True)          # (B, L+1, V)
old_logp = torch.tensor([[-math.log(2.0) - math.log(1.4)]], device=dev)
ids = torch.tensor([[0]], device=dev)
adv = torch.tensor([-1.0], device=dev)

loss, _ = triton_grpo_loss(logits, old_logp, None, ids, adv, temperature=1.0, beta=0.0,
                           eps_low=0.25, eps_high=0.5, inplace=False, loss_type="grpo",
                           reduce=True, delta=1.25)
loss.backward()

# Reference (TRL semantics): coef_2 = clamp(ratio, 1-eps_low, 1+eps_high); coef_1 = clamp(ratio, max=delta)
ref_logits = logits.detach().clone().requires_grad_()
ratio = (ref_logits[:, :-1].log_softmax(-1)[..., 0] - old_logp).exp()
ref = -torch.minimum(ratio.clamp(max=1.25) * adv, ratio.clamp(0.75, 1.5) * adv).mean()
ref.backward()

print("loss:", loss.item(), "reference:", ref.item())
print("grad:", logits.grad[0, 0].tolist(), "reference:", ref_logits.grad[0, 0].tolist())

Output:

loss: 1.399999976158142 reference: 1.4000000953674316
grad: [0.0, 0.0] reference: [0.7000000476837158, -0.7000000476837158]
Expected behavior

loss 1.4 (correct now) and gradient [0.7, -0.7] (TRL semantics, as in torch_grpo_loss in test/transformers/test_grpo_loss.py).

Suggested fix

When branch 2 wins, use dlogp = -coef_1 * advantage if 1 - eps_low <= coef_1 <= 1 + eps_high, else 0 (token and sequence level).

Versions / environment
  • Liger-Kernel 83746f4 (the lines cited above are unchanged on main as of 2026-11-17).
  • torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
  • The root cause is a branch condition in the kernel, so it is the same on CUDA. The script runs on cuda when it is available.
Dominant language
Python
Stars
6.7k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Liger-Kernel

All issues in linkedin/Liger-Kernel

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.