# [Bug] Multi-token attention mask kernels overflow int32 batch offsets (`batch_id * stride_b`) past 2^31

Aperta
#1,659 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Read src/liger_kernel/ops/multi_token_attention.py, focusing on _mask_fwd_kernel and _mask_bwd_kernel and the four wrappers named in the issue. Reproduce the reported overflow case and check how widening batch_id affects both kernels. Done means forward and backward wrappers handle offsets beyond 2^31 elements without illegal memory access and the existing cases still pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐛 Describe the bug
🐛 Describe the bug

_mask_fwd_kernel and _mask_bwd_kernel in multi_token_attention.py compute the start of each L x L matrix as batch_id * stride_b. batch_id is tl.program_id(2) and stride_b is passed as an i32, so the product is int32 (mul.lo.s32 in the PTX). The wrappers pass the strides of the contiguous (N, L, L) view, so stride_b = L * L. The first matrix whose offset reaches 2^31 elements (batch_id * L * L >= 2**31) wraps to a negative offset, and the kernel loads and stores before the tensor. In practice this is an illegal memory access.

All four wrappers launch these kernels (_mask_inf_forward, _mask_zero_forward, _mask_zero_backward, _mask_inf_backward). LigerMultiTokenAttentionFunction reaches them in both forward and backward whenever batch * heads * L * L > 2**31. Nothing on the host side rejects such a shape: the wrappers check neither N nor the product. For example, at L = 1024, batch * heads = 2049 is enough (4 GiB in bf16).

This is the same class of 32-bit offset arithmetic as #804 and #1483 (RMSNorm's program_id * stride), and it is still the same on current main (781a8ae):

Reproduce

The module's own wrappers on one (N, L, L) tensor. To fit both the input and the output in 8 GB, the tensor is int8 (the offsets are in elements, so bf16 overflows at the same N, with 4 GiB per tensor). L = 256, so 2**31 elements is N = 32768:

import torch
from liger_kernel.ops.multi_token_attention import _mask_zero_backward, _mask_zero_forward

N, L = 32769, 256                      # N * L * L = 2**31 + 65536
x = torch.ones((N, L, L), device="cuda", dtype=torch.int8)
out = _mask_zero_backward(x)           # same with _mask_zero_forward(x)
torch.cuda.synchronize()
# torch.AcceleratorError: CUDA error: an illegal memory access was encountered

With N = 32768 (exactly 2**31 elements, the last matrix's offset 2**31 - 65536) both wrappers return tril(ones) for every matrix. Each case ran in its own process:

wrapper N = 32768 N = 32769 N = 32769, batch_id int64
_mask_zero_backward (_mask_bwd_kernel) ✅ 0 of 32768 matrices wrong ❌ illegal memory access ✅ 0 of 32769 matrices wrong
_mask_zero_forward (_mask_fwd_kernel) ✅ 0 of 32768 matrices wrong ❌ illegal memory access ✅ 0 of 32769 matrices wrong

The fix is to widen batch_id before the multiply, in both kernels:

-    batch_id = tl.program_id(2)
+    batch_id = tl.program_id(2).to(tl.int64)

The patch is attached. offs = row_idx * stride_m + col_idx * stride_n stays int32. It is below L * L and does not overflow for any L whose BLOCK x BLOCK tile compiles.

Versions

torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Reproduced with the kernels at Liger-Kernel 488acbd; the two kernels are unchanged on main @ 781a8ae.

Lingua principale
Python
Stelle
6.7k
Fork
668
Merge medio
4g 12h
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di linkedin/Liger-Kernel

Tutte le issue di linkedin/Liger-Kernel

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.