triton_grpo_loss: zero gradient with delta when the unclipped branch wins (negative advantage)

Aperta
#1,643 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@shotsan ci sta già lavorando.

Dal 18/11/2026.

  • #1655 di @shotsan — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
77/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch

Direzione di ricerca

Read src/liger_kernel/ops/grpo_loss.py around the token and sequence level backward logic at the cited lines, then compare its behavior with torch_grpo_loss in test/transformers/test_grpo_loss.py. Run the provided reproduction in Triton's interpreter mode and check both token and sequence cases. Done means the loss remains correct and the gradient matches the reference for negative advantages when delta is set.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

With delta set (two-sided clipping), the loss is -min(min(r, delta) * A, clip(r, 1-eps_low, 1+eps_high) * A). When A < 0 and delta < r < 1 + eps_high, min selects the second branch. In that branch r is not clipped, so the gradient is -r * A * dlogp, which is not zero. The kernel returns a zero gradient. The loss value is correct.

Cause: ops/grpo_loss.py#L482-L486:

mask = per_token_loss2 >= per_token_loss1
dlogp = -coef_1 * advantage * mask          # gradient only if branch 1 wins
if DELTA != 0.0:
    dlogp = dlogp * (coef_1 <= DELTA)

This assumes that "branch 2 wins" means "coef_2 is clipped". That is true without delta, but with delta, branch 2 can win while coef_2 = coef_1 is inside the clip range. The sequence-level backward has the same logic (#L377-L383).

Reproduce
import math
import torch
from liger_kernel.transformers.grpo_loss import triton_grpo_loss

dev = "cuda" if torch.cuda.is_available() else "cpu"
# 1 sequence, 1 completion token, vocab 2. log p(token) = log(1/2).
# old_logp is chosen so that ratio = p / p_old = 1.4:
#   1.4 > delta = 1.25, and 1.4 is inside [1 - eps_low, 1 + eps_high] = [0.75, 1.5]. Advantage = -1.
logits = torch.zeros(1, 2, 2, device=dev, requires_grad=True)          # (B, L+1, V)
old_logp = torch.tensor([[-math.log(2.0) - math.log(1.4)]], device=dev)
ids = torch.tensor([[0]], device=dev)
adv = torch.tensor([-1.0], device=dev)

loss, _ = triton_grpo_loss(logits, old_logp, None, ids, adv, temperature=1.0, beta=0.0,
                           eps_low=0.25, eps_high=0.5, inplace=False, loss_type="grpo",
                           reduce=True, delta=1.25)
loss.backward()

# Reference (TRL semantics): coef_2 = clamp(ratio, 1-eps_low, 1+eps_high); coef_1 = clamp(ratio, max=delta)
ref_logits = logits.detach().clone().requires_grad_()
ratio = (ref_logits[:, :-1].log_softmax(-1)[..., 0] - old_logp).exp()
ref = -torch.minimum(ratio.clamp(max=1.25) * adv, ratio.clamp(0.75, 1.5) * adv).mean()
ref.backward()

print("loss:", loss.item(), "reference:", ref.item())
print("grad:", logits.grad[0, 0].tolist(), "reference:", ref_logits.grad[0, 0].tolist())

Output:

loss: 1.399999976158142 reference: 1.4000000953674316
grad: [0.0, 0.0] reference: [0.7000000476837158, -0.7000000476837158]
Expected behavior

loss 1.4 (correct now) and gradient [0.7, -0.7] (TRL semantics, as in torch_grpo_loss in test/transformers/test_grpo_loss.py).

Suggested fix

When branch 2 wins, use dlogp = -coef_1 * advantage if 1 - eps_low <= coef_1 <= 1 + eps_high, else 0 (token and sequence level).

Versions / environment
  • Liger-Kernel 83746f4 (the lines cited above are unchanged on main as of 2026-11-17).
  • torch 2.14.1, triton 3.8.0. Reproduced with Triton's interpreter mode (TRITON_INTERPRET=1).
  • The root cause is a branch condition in the kernel, so it is the same on CUDA. The script runs on cuda when it is available.
Lingua principale
Python
Stelle
6.7k
Fork
668
Merge medio
4g 12h
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di linkedin/Liger-Kernel

Tutte le issue di linkedin/Liger-Kernel

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.