[Bug] Vocab-parallel CE treats every target of the rank whose shard crosses 2^31 as off-rank (`vocab_start + n_cols` wraps in int32)

Open
#1,661 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

Read src/liger_kernel/ops/vocab_parallel_cross_entropy.py, focusing on the forward and backward kernels near the reported target ownership checks and how vocab_start is passed. Start by checking the issue’s reproducer and expected results; then run the relevant vocab parallel cross entropy tests. Done means targets in a shard crossing 2^31 produce the expected loss and gradient in both kernels.

Written by the indexing model from the issue text.

Description

🐛 Describe the bug

liger_vocab_parallel_ce_forward_kernel and liger_vocab_parallel_ce_backward_kernel decide whether a row's target belongs to this rank with

target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)

vocab_start = rank * v_local and n_cols = v_local are Python ints, and Triton passes an int argument as i32 when it fits (specialize_impl). So vocab_start + n_cols is added in int32 and only then widened for the comparison with the int64 target. On the rank whose shard starts below 2^31 and ends at or past it, the shard end wraps to a negative number. Then y_global >= vocab_start + n_cols holds for every target, and the rank treats all of its own targets as off-rank:

  • Forward stores a predicted logit of 0 instead of X[y_local] - logits_max. No other rank owns the target either, so after the AllReduce the loss is log(sum_exp) instead of log(sum_exp) - (X[y] - logits_max).
  • Backward leaves out the target's -(1 - alpha_eff), so the target's gradient is p instead of p - 1.

Ranks whose vocab_start is at least 2^31 are fine: Triton passes vocab_start as an i64, and the sum is int64.

To be upfront: this needs a global vocabulary of at least 2^31 (e.g. TP=8 with 2^28 logits per rank), far beyond today's vocabularies, so it is unlikely to happen in practice. It is the same 32-bit arithmetic as #1570 and #1573. Nothing rejects such a shape, and nothing reports an error; the loss and the gradient are just wrong.

On current main (673ab11):

Reproduce

A TP=8 run with 2^28 logits per rank does not fit on one GPU, so this launches the two kernels the way LigerVocabParallelCEFunction launches them on one rank (vocab_start = rank * v_local, n_cols = v_local), on one row whose target is the rank's first logit:

import torch
from liger_kernel.ops.vocab_parallel_cross_entropy import (
    _get_num_warps,
    _select_block_size,
    liger_vocab_parallel_ce_backward_kernel,
    liger_vocab_parallel_ce_forward_kernel,
)


def run(rank, v_local):
    # one row on one rank, launched as LigerVocabParallelCEFunction launches it
    vocab_start = rank * v_local
    x = torch.zeros((1, v_local), device="cuda", dtype=torch.bfloat16)
    x[0, 0], x[0, 1] = 1.0, 2.0
    target = torch.tensor([vocab_start], device="cuda")  # this rank's first logit
    logits_max = x.amax(dim=-1).float()
    exp_buf = torch.empty((1, v_local), device="cuda", dtype=torch.float32)
    pred = torch.empty(1, device="cuda")
    sum_exp = torch.empty(1, device="cuda")
    block_size = _select_block_size(v_local)
    num_warps = _get_num_warps(block_size)
    liger_vocab_parallel_ce_forward_kernel[(1,)](
        X_ptr=x, X_stride=x.stride(0), EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0),
        logits_max_ptr=logits_max, Y_ptr=target, pred_ptr=pred, sum_exp_ptr=sum_exp,
        vocab_start=vocab_start, n_cols=v_local, ignore_index=-100,
        BLOCK_SIZE=block_size, HAS_LABEL_SMOOTHING=False, num_warps=num_warps,
    )
    p_target = (exp_buf[0, 0] / sum_exp[0]).item()
    liger_vocab_parallel_ce_backward_kernel[(1,)](
        EXP_ptr=exp_buf, EXP_stride=exp_buf.stride(0), sum_exp_ptr=sum_exp, Y_ptr=target,
        grad_out_ptr=torch.ones(1, device="cuda"), vocab_start=vocab_start, n_cols=v_local,
        ignore_index=-100, alpha_eff=0.0, eps_eff=0.0,
        HAS_LABEL_SMOOTHING=False, BLOCK_SIZE=block_size, num_warps=num_warps,
    )
    torch.cuda.synchronize()
    print(f"rank {rank}, v_local {v_local}, shard [{vocab_start}, {vocab_start + v_local}): "
          f"predicted logit {pred.item():.4f} (expected {(x[0, 0] - logits_max[0]).item():.4f}), "
          f"target grad {exp_buf[0, 0].item():.4f} (expected {p_target - 1:.4f})")


run(7, 2**28)      # last rank of TP=8, V = 2**31
run(7, 2**28 - 1)  # last rank of TP=8, V = 2**31 - 8
run(8, 2**28)      # last rank of TP=9, V = 2**31 + 2**28
rank 7, v_local 268435456, shard [1879048192, 2147483648): predicted logit 0.0000 (expected -1.0000), target grad 0.0000 (expected -1.0000)
rank 7, v_local 268435455, shard [1879048185, 2147483640): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank 8, v_local 268435456, shard [2147483648, 2415919104): predicted logit -1.0000 (expected -1.0000), target grad -1.0000 (expected -1.0000)
rank, v_local shard main vocab_start widened
7, 2**28 (TP=8) [1879048192, 2147483648) ❌ predicted logit 0, target grad 0 ✅ -1, -1
7, 2**28 - 1 (TP=8) [1879048185, 2147483640) ✅ -1, -1 ✅ -1, -1
8, 2**28 (TP=9) [2147483648, 2415919104) ✅ -1, -1 ✅ -1, -1

The fix is to widen vocab_start before the addition, in both kernels:

-    target_off_rank = (y_global < vocab_start) | (y_global >= vocab_start + n_cols)
+    target_off_rank = (y_global < vocab_start) | (y_global >= tl.cast(vocab_start, tl.int64) + n_cols)

The patch is attached.

Versions

torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Liger-Kernel main @ 673ab11.

Dominant language
Python
Stars
6.7k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Liger-Kernel

All issues in linkedin/Liger-Kernel

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.