[Bug] Vocab-parallel CE treats every target of the rank whose shard crosses 2^31 as off-rank (`vocab_start + n_cols` wraps in int32)

Aperta
#1,661 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
68/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch

Direzione di ricerca

Read src/liger_kernel/ops/vocab_parallel_cross_entropy.py, focusing on the forward and backward kernels near the reported target ownership checks and how vocab_start is passed. Start by checking the issue’s reproducer and expected results; then run the relevant vocab parallel cross entropy tests. Done means targets in a shard crossing 2^31 produce the expected loss and gradient in both kernels.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐛 Describe the bug

liger_vocab_parallel_ce_forward_kernel and liger_vocab_parallel_ce_backward_kernel decide whether a row's target belongs to this rank with

target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)

vocab_start = rank * v_local and n_cols = v_local are Python ints, and Triton passes an int argument as i32 when it fits (specialize_impl). So vocab_start + n_cols is added in int32 and only then widened for the comparison with the int64 target. On the rank whose shard starts below 2^31 and ends at or past it, the shard end wraps to a negative number. Then y_global >= vocab_start + n_cols holds for every target, and the rank treats all of its own targets as off-rank:

  • Forward stores a predicted logit of 0 instead of X[y_local] - logits_max. No other rank owns the target either, so after the AllReduce the loss is log(sum_exp) instead of log(sum_exp) - (X[y] - logits_max).
  • Backward leaves out the target's -(1 - alpha_eff), so the target's gradient is p instead of p - 1.

Ranks whose vocab_start is at least 2^31 are fine: Triton passes vocab_start as an i64, and the sum is int64.

To be upfront: this needs a global vocabulary of at least 2^31 (e.g. TP=8 with 2^28 logits per rank), far beyond today's vocabularies, so it is unlikely to happen in practice. It is the same 32-bit arithmetic as #1570 and #1573. Nothing rejects such a shape, and nothing reports an error; the loss and the gradient are just wrong.

On current main (673ab11):

Reproduce

A TP=8 run with 2^28 logits per rank does not fit on one GPU, so this launches the two kernels the way LigerVocabParallelCEFunction launches them on one rank (vocab_start = rank * v_local, n_cols = v_local), on one row whose target is the rank's first logit:

import torch
from liger_kernel.ops.vocab_parallel_cross_entropy import (
    _get_num_warps,
    _select_block_size,
    liger_vocab_parallel_ce_backward_kernel,
    liger_vocab_parallel_ce_forward_kernel,
)


def run(rank, v_local):
    # one row on one rank, launched as LigerVocabParallelCEFunction launches it
    vocab_start = rank * v_local
    x = torch.zeros((1, v_local), device="cuda", dtype=torch.bfloat16)
    x[0, 0], x[0, 1] = 1.0, 2.0
    target = torch.tensor([vocab_start], device="cuda")  # this rank's first logit
    logits_max = x.amax(dim=-1).float()
    exp_buf = torch.empty((1, v_local), device="cuda", dtype=torch.float32)
    pred = torch.empty(1, device="cuda")
    sum_exp = torch.empty(1, device="cuda")
    block_size = _select_block_size(v_local)
    num_warps = _get_num_warps(block_size)
    liger_vocab_parallel_ce_forward_kernel[(1,)](
        X_ptr=x, X_stride=x.stride(0), EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0),
        logits_max_ptr=logits_max, Y_ptr=target, pred_ptr=pred, sum_exp_ptr=sum_exp,
        vocab_start=vocab_start, n_cols=v_local, ignore_index=-100,
        BLOCK_SIZE=block_size, HAS_LABEL_SMOOTHING=False, num_warps=num_warps,
    )
    p_target = (exp_buf[0, 0] / sum_exp[0]).item()
    liger_vocab_parallel_ce_backward_kernel[(1,)](
        EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0), sum_exp_ptr=sum_exp, Y_ptr=target,
        grad_out_ptr=torch.ones(1, device="cuda"), vocab_start=vocab_start, n_cols=v_local,
        ignore_index=-100, alpha_eff=0.0, eps_eff=0.0,
        HAS_LABEL_SMOOTHING=False, BLOCK_SIZE=block_size, num_warps=num_warps,
    )
    torch.cuda.synchronize()
    print(f"rank {rank}, v_local {v_local}, shard [{vocab_start}, {vocab_start + v_local}): "
          f"predicted logit {pred.item():.4f} (expected {(x[0, 0] - logits_max[0]).item():.4f}), "
          f"target grad {exp_buf[0, 0].item():.4f} (expected {p_target - 1:.4f})")


run(7, 2**28)      # last rank of TP=8, V = 2**31
run(7, 2**28 - 1)  # last rank of TP=8, V = 2**31 - 8
run(8, 2**28)      # last rank of TP=9, V = 2**31 + 2**28
rank 7, v_local 268435456, shard [1879048192, 2147483648): predicted logit 0.0000 (expected -1.0000), target grad 0.0000 (expected -1.0000)
rank 7, v_local 268435455, shard [1879048185, 2147483640): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank 8, v_local 268435456, shard [2147483648, 2415919104): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank, v_local shard main vocab_start widened
7, 2**28 (TP=8) [1879048192, 2147483648) ❌ predicted logit 0, target grad 0 ✅ -1, -1
7, 2**28 - 1 (TP=8) [1879048185, 2147483640) ✅ -1, -1 ✅ -1, -1
8, 2**28 (TP=9) [2147483648, 2415919104) ✅ -1, -1 ✅ -1, -1

The fix is to widen vocab_start before the addition, in both kernels:

-    target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)
+    target_off_rank = (y_global < vocab_start) | (y_global >= tl.cast(vocab_start, tl.int64) + n_cols)

The patch is attached.

Versions

torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Liger-Kernel main @ 673ab11.

Lingua principale
Python
Stelle
6.7k
Fork
668
Merge medio
4g 12h
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di linkedin/Liger-Kernel

Tutte le issue di linkedin/Liger-Kernel

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.