[Bug] Vocab-parallel CE treats every target of the rank whose shard crosses 2^31 as off-rank (`vocab_start + n_cols` wraps in int32)
I maintainer di solito rispondono entro 3 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 68/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Ambito
- machine-learning, performance
Direzione di ricerca
Read src/liger_kernel/ops/vocab_parallel_cross_entropy.py, focusing on the forward and backward kernels near the reported target ownership checks and how vocab_start is passed. Start by checking the issue’s reproducer and expected results; then run the relevant vocab parallel cross entropy tests. Done means targets in a shard crossing 2^31 produce the expected loss and gradient in both kernels.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
🐛 Describe the bug
liger_vocab_parallel_ce_forward_kernel and liger_vocab_parallel_ce_backward_kernel decide whether a row's target belongs to this rank with
target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)
vocab_start = rank * v_local and n_cols = v_local are Python ints, and Triton passes an int argument as i32 when it fits (specialize_impl). So vocab_start + n_cols is added in int32 and only then widened for the comparison with the int64 target. On the rank whose shard starts below 2^31 and ends at or past it, the shard end wraps to a negative number. Then y_global >= vocab_start + n_cols holds for every target, and the rank treats all of its own targets as off-rank:
- Forward stores a predicted logit of 0 instead of
X[y_local] - logits_max. No other rank owns the target either, so after the AllReduce the loss islog(sum_exp)instead oflog(sum_exp) - (X[y] - logits_max). - Backward leaves out the target's
-(1 - alpha_eff), so the target's gradient ispinstead ofp - 1.
Ranks whose vocab_start is at least 2^31 are fine: Triton passes vocab_start as an i64, and the sum is int64.
To be upfront: this needs a global vocabulary of at least 2^31 (e.g. TP=8 with 2^28 logits per rank), far beyond today's vocabularies, so it is unlikely to happen in practice. It is the same 32-bit arithmetic as #1570 and #1573. Nothing rejects such a shape, and nothing reports an error; the loss and the gradient are just wrong.
On current main (673ab11):
liger_vocab_parallel_ce_forward_kernelL94liger_vocab_parallel_ce_backward_kernelL154vocab_startis computed inLigerVocabParallelCEFunction.forwardL242
Reproduce
A TP=8 run with 2^28 logits per rank does not fit on one GPU, so this launches the two kernels the way LigerVocabParallelCEFunction launches them on one rank (vocab_start = rank * v_local, n_cols = v_local), on one row whose target is the rank's first logit:
import torch
from liger_kernel.ops.vocab_parallel_cross_entropy import (
_get_num_warps,
_select_block_size,
liger_vocab_parallel_ce_backward_kernel,
liger_vocab_parallel_ce_forward_kernel,
)
def run(rank, v_local):
# one row on one rank, launched as LigerVocabParallelCEFunction launches it
vocab_start = rank * v_local
x = torch.zeros((1, v_local), device="cuda", dtype=torch.bfloat16)
x[0, 0], x[0, 1] = 1.0, 2.0
target = torch.tensor([vocab_start], device="cuda") # this rank's first logit
logits_max = x.amax(dim=-1).float()
exp_buf = torch.empty((1, v_local), device="cuda", dtype=torch.float32)
pred = torch.empty(1, device="cuda")
sum_exp = torch.empty(1, device="cuda")
block_size = _select_block_size(v_local)
num_warps = _get_num_warps(block_size)
liger_vocab_parallel_ce_forward_kernel[(1,)](
X_ptr=x, X_stride=x.stride(0), EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0),
logits_max_ptr=logits_max, Y_ptr=target, pred_ptr=pred, sum_exp_ptr=sum_exp,
vocab_start=vocab_start, n_cols=v_local, ignore_index=-100,
BLOCK_SIZE=block_size, HAS_LABEL_SMOOTHING=False, num_warps=num_warps,
)
p_target = (exp_buf[0, 0] / sum_exp[0]).item()
liger_vocab_parallel_ce_backward_kernel[(1,)](
EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0), sum_exp_ptr=sum_exp, Y_ptr=target,
grad_out_ptr=torch.ones(1, device="cuda"), vocab_start=vocab_start, n_cols=v_local,
ignore_index=-100, alpha_eff=0.0, eps_eff=0.0,
HAS_LABEL_SMOOTHING=False, BLOCK_SIZE=block_size, num_warps=num_warps,
)
torch.cuda.synchronize()
print(f"rank {rank}, v_local {v_local}, shard [{vocab_start}, {vocab_start + v_local}): "
f"predicted logit {pred.item():.4f} (expected {(x[0, 0] - logits_max[0]).item():.4f}), "
f"target grad {exp_buf[0, 0].item():.4f} (expected {p_target - 1:.4f})")
run(7, 2**28) # last rank of TP=8, V = 2**31
run(7, 2**28 - 1) # last rank of TP=8, V = 2**31 - 8
run(8, 2**28) # last rank of TP=9, V = 2**31 + 2**28
rank 7, v_local 268435456, shard [1879048192, 2147483648): predicted logit 0.0000 (expected -1.0000), target grad 0.0000 (expected -1.0000)
rank 7, v_local 268435455, shard [1879048185, 2147483640): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank 8, v_local 268435456, shard [2147483648, 2415919104): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank, v_local |
shard | main |
vocab_start widened |
|---|---|---|---|
7, 2**28 (TP=8) |
[1879048192, 2147483648) |
❌ predicted logit 0, target grad 0 | ✅ -1, -1 |
7, 2**28 - 1 (TP=8) |
[1879048185, 2147483640) |
✅ -1, -1 | ✅ -1, -1 |
8, 2**28 (TP=9) |
[2147483648, 2415919104) |
✅ -1, -1 | ✅ -1, -1 |
The fix is to widen vocab_start before the addition, in both kernels:
- target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)
+ target_off_rank = (y_global < vocab_start) | (y_global >= tl.cast(vocab_start, tl.int64) + n_cols)
The patch is attached.
Versions
torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Liger-Kernel main @ 673ab11.
- Lingua principale
- Python
- Stelle
- 6.7k
- Fork
- 668
- Merge medio
- 4g 12h
- PR unite (30g)
- 27
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di linkedin/Liger-Kernel
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
linkedin/Liger-Kernel#1658 ·
I maintainer di solito rispondono entro 3 giorni
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsForse già presa @nazek52 l’ha presa 1 giorno fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
linkedin/Liger-Kernel#1647 ·
I maintainer di solito rispondono entro 3 giorni
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Forse già presa @0xDexe l’ha presa 8 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
linkedin/Liger-Kernel#1570 · 3 commenti ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 55/100
linkedin/Liger-Kernel#1660 ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 45/100
linkedin/Liger-Kernel#1659 ·
I maintainer di solito rispondono entro 3 giorni
Tutte le issue di linkedin/Liger-Kernel
Issue simili
-
Internal ErrorApertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
frappe/frappe_docker#2071 ·
I maintainer di solito rispondono entro 1 giorno
-
bug python
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
traceloop/openllmetry#4717 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
The-Pocket/PocketFlow#161 ·
-
[Bug]: Webapp workspace fails to render in Safari — "Can't find variable: requestIdleCallback"Apertacomp/desktop P2 type/bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
NousResearch/hermes-agent#157702 ·
I maintainer di solito rispondono entro 1 giorno
-
scibench_physics / scibench_chemistry: a correct answer scores 0 when the unit has a power of tenForse già presa @BahaaMattar l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
EvolvingLMMs-Lab/lmms-eval#1699 ·
I maintainer di solito rispondono entro 1 giorno