[Bug] Vocab-parallel CE treats every target of the rank whose shard crosses 2^31 as off-rank (`vocab_start + n_cols` wraps in int32)
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- machine-learning, performance
Research direction
Read src/liger_kernel/ops/vocab_parallel_cross_entropy.py, focusing on the forward and backward kernels near the reported target ownership checks and how vocab_start is passed. Start by checking the issue’s reproducer and expected results; then run the relevant vocab parallel cross entropy tests. Done means targets in a shard crossing 2^31 produce the expected loss and gradient in both kernels.
Written by the indexing model from the issue text.
Description
🐛 Describe the bug
liger_vocab_parallel_ce_forward_kernel and liger_vocab_parallel_ce_backward_kernel decide whether a row's target belongs to this rank with
target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)
vocab_start = rank * v_local and n_cols = v_local are Python ints, and Triton passes an int argument as i32 when it fits (specialize_impl). So vocab_start + n_cols is added in int32 and only then widened for the comparison with the int64 target. On the rank whose shard starts below 2^31 and ends at or past it, the shard end wraps to a negative number. Then y_global >= vocab_start + n_cols holds for every target, and the rank treats all of its own targets as off-rank:
- Forward stores a predicted logit of 0 instead of
X[y_local] - logits_max. No other rank owns the target either, so after the AllReduce the loss islog(sum_exp)instead oflog(sum_exp) - (X[y] - logits_max). - Backward leaves out the target's
-(1 - alpha_eff), so the target's gradient ispinstead ofp - 1.
Ranks whose vocab_start is at least 2^31 are fine: Triton passes vocab_start as an i64, and the sum is int64.
To be upfront: this needs a global vocabulary of at least 2^31 (e.g. TP=8 with 2^28 logits per rank), far beyond today's vocabularies, so it is unlikely to happen in practice. It is the same 32-bit arithmetic as #1570 and #1573. Nothing rejects such a shape, and nothing reports an error; the loss and the gradient are just wrong.
On current main (673ab11):
liger_vocab_parallel_ce_forward_kernelL94liger_vocab_parallel_ce_backward_kernelL154vocab_startis computed inLigerVocabParallelCEFunction.forwardL242
Reproduce
A TP=8 run with 2^28 logits per rank does not fit on one GPU, so this launches the two kernels the way LigerVocabParallelCEFunction launches them on one rank (vocab_start = rank * v_local, n_cols = v_local), on one row whose target is the rank's first logit:
import torch
from liger_kernel.ops.vocab_parallel_cross_entropy import (
_get_num_warps,
_select_block_size,
liger_vocab_parallel_ce_backward_kernel,
liger_vocab_parallel_ce_forward_kernel,
)
def run(rank, v_local):
# one row on one rank, launched as LigerVocabParallelCEFunction launches it
vocab_start = rank * v_local
x = torch.zeros((1, v_local), device="cuda", dtype=torch.bfloat16)
x[0, 0], x[0, 1] = 1.0, 2.0
target = torch.tensor([vocab_start], device="cuda") # this rank's first logit
logits_max = x.amax(dim=-1).float()
exp_buf = torch.empty((1, v_local), device="cuda", dtype=torch.float32)
pred = torch.empty(1, device="cuda")
sum_exp = torch.empty(1, device="cuda")
block_size = _select_block_size(v_local)
num_warps = _get_num_warps(block_size)
liger_vocab_parallel_ce_forward_kernel[(1,)](
X_ptr=x, X_stride=x.stride(0), EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0),
logits_max_ptr=logits_max, Y_ptr=target, pred_ptr=pred, sum_exp_ptr=sum_exp,
vocab_start=vocab_start, n_cols=v_local, ignore_index=-100,
BLOCK_SIZE=block_size, HAS_LABEL_SMOOTHING=False, num_warps=num_warps,
)
p_target = (exp_buf[0, 0] / sum_exp[0]).item()
liger_vocab_parallel_ce_backward_kernel[(1,)](
EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0), sum_exp_ptr=sum_exp, Y_ptr=target,
grad_out_ptr=torch.ones(1, device="cuda"), vocab_start=vocab_start, n_cols=v_local,
ignore_index=-100, alpha_eff=0.0, eps_eff=0.0,
HAS_LABEL_SMOOTHING=False, BLOCK_SIZE=block_size, num_warps=num_warps,
)
torch.cuda.synchronize()
print(f"rank {rank}, v_local {v_local}, shard [{vocab_start}, {vocab_start + v_local}): "
f"predicted logit {pred.item():.4f} (expected {(x[0, 0] - logits_max[0]).item():.4f}), "
f"target grad {exp_buf[0, 0].item():.4f} (expected {p_target - 1:.4f})")
run(7, 2**28) # last rank of TP=8, V = 2**31
run(7, 2**28 - 1) # last rank of TP=8, V = 2**31 - 8
run(8, 2**28) # last rank of TP=9, V = 2**31 + 2**28
rank 7, v_local 268435456, shard [1879048192, 2147483648): predicted logit 0.0000 (expected -1.0000), target grad 0.0000 (expected -1.0000)
rank 7, v_local 268435455, shard [1879048185, 2147483640): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank 8, v_local 268435456, shard [2147483648, 2415919104): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank, v_local |
shard | main |
vocab_start widened |
|---|---|---|---|
7, 2**28 (TP=8) |
[1879048192, 2147483648) |
❌ predicted logit 0, target grad 0 | ✅ -1, -1 |
7, 2**28 - 1 (TP=8) |
[1879048185, 2147483640) |
✅ -1, -1 | ✅ -1, -1 |
8, 2**28 (TP=9) |
[2147483648, 2415919104) |
✅ -1, -1 | ✅ -1, -1 |
The fix is to widen vocab_start before the addition, in both kernels:
- target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)
+ target_off_rank = (y_global < vocab_start) | (y_global >= tl.cast(vocab_start, tl.int64) + n_cols)
The patch is attached.
Versions
torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Liger-Kernel main @ 673ab11.
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
linkedin/Liger-Kernel#1659 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day