triton_grpo_loss: zero gradient with delta when the unclipped branch wins (negative advantage)
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 77/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- machine-learning, testing
Research direction
Read src/liger_kernel/ops/grpo_loss.py around the token and sequence level backward logic at the cited lines, then compare its behavior with torch_grpo_loss in test/transformers/test_grpo_loss.py. Run the provided reproduction in Triton's interpreter mode and check both token and sequence cases. Done means the loss remains correct and the gradient matches the reference for negative advantages when delta is set.
Written by the indexing model from the issue text.
Description
Describe the bug
With delta set (two-sided clipping), the loss is -min(min(r, delta) * A, clip(r, 1-eps_low, 1+eps_high) * A). When A < 0 and delta < r < 1 + eps_high, min selects the second branch. In that branch r is not clipped, so the gradient is -r * A * dlogp, which is not zero. The kernel returns a zero gradient. The loss value is correct.
Cause: ops/grpo_loss.py#L482-L486:
mask = per_token_loss2 >= per_token_loss1
dlogp = -coef_1 * advantage * mask # gradient only if branch 1 wins
if DELTA != 0.0:
dlogp = dlogp * (coef_1 <= DELTA)
This assumes that "branch 2 wins" means "coef_2 is clipped". That is true without delta, but with delta, branch 2 can win while coef_2 = coef_1 is inside the clip range. The sequence-level backward has the same logic (#L377-L383).
Reproduce
import math
import torch
from liger_kernel.transformers.grpo_loss import triton_grpo_loss
dev = "cuda" if torch.cuda.is_available() else "cpu"
# 1 sequence, 1 completion token, vocab 2. log p(token) = log(1/2).
# old_logp is chosen so that ratio = p / p_old = 1.4:
# 1.4 > delta = 1.25, and 1.4 is inside [1 - eps_low, 1 + eps_high] = [0.75, 1.5]. Advantage = -1.
logits = torch.zeros(1, 2, 2, device=dev, requires_grad=True) # (B, L+1, V)
old_logp = torch.tensor([[-math.log(2.0) - math.log(1.4)]], device=dev)
ids = torch.tensor([[0]], device=dev)
adv = torch.tensor([-1.0], device=dev)
loss, _ = triton_grpo_loss(logits, old_logp, None, ids, adv, temperature=1.0, beta=0.0,
eps_low=0.25, eps_high=0.5, inplace=False, loss_type="grpo",
reduce=True, delta=1.25)
loss.backward()
# Reference (TRL semantics): coef_2 = clamp(ratio, 1-eps_low, 1+eps_high); coef_1 = clamp(ratio, max=delta)
ref_logits = logits.detach().clone().requires_grad_()
ratio = (ref_logits[:, :-1].log_softmax(-1)[..., 0] - old_logp).exp()
ref = -torch.minimum(ratio.clamp(max=1.25) * adv, ratio.clamp(0.75, 1.5) * adv).mean()
ref.backward()
print("loss:", loss.item(), "reference:", ref.item())
print("grad:", logits.grad[0, 0].tolist(), "reference:", ref_logits.grad[0, 0].tolist())
Output:
loss: 1.399999976158142 reference: 1.4000000953674316
grad: [0.0, 0.0] reference: [0.7000000476837158, -0.7000000476837158]
Expected behavior
loss 1.4 (correct now) and gradient [0.7, -0.7] (TRL semantics, as in torch_grpo_loss in test/transformers/test_grpo_loss.py).
Suggested fix
When branch 2 wins, use dlogp = -coef_1 * advantage if 1 - eps_low <= coef_1 <= 1 + eps_high, else 0 (token and sequence level).
Versions / environment
- Liger-Kernel
83746f4(the lines cited above are unchanged onmainas of 2026-11-17). - torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (
TRITON_INTERPRET=1). - The root cause is a branch condition in the kernel, so it is the same on CUDA. The script runs on
cudawhen it is available.
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day