# [Bug] `_l2_swizzle` in mlp.py counts tiles in int32 (wrong tile mapping when `S` or `hidden_dim` is within a block of 2^31)
I maintainer di solito rispondono entro 3 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Idoneità per principianti
- 77/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- python
- Ambito
- machine-learning, performance
Direzione di ricerca
Leggi src/liger_kernel/ops/mlp.py, in particolare _l2_swizzle alla riga 33 e i quattro punti di chiamata del kernel SwiGLU elencati nell’issue. Inizia verificando il calcolo del numero di tile rispetto al riferimento con interi Python dello script di riproduzione, per dimensioni vicine a 2^31. Il lavoro è completato quando lo swizzle mappa ogni ID di programma testato sul tile di riferimento, comprese le quattro forme limite descritte.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
🐛 Describe the bug
_l2_swizzle in mlp.py:33 maps a program id to its output tile (pid_m, pid_n). It starts from
num_pid_m = tl.cdiv(M, BLOCK_M)
num_pid_n = tl.cdiv(N, BLOCK_N)
where M and N are the kernel's int32 shape arguments. tl.cdiv(x, d) is (x + d - 1) // d, so once M > 2**31 - BLOCK_M (or N > 2**31 - BLOCK_N) the sum wraps to a negative number. The group arithmetic below it then hands out the wrong tiles: some tiles are computed twice, others never, and some programs get a negative pid_m, so their descriptor loads and stores fall outside the tensor.
The SwiGLU MLP kernels all go through it:
_swiglu_kernel_forwardL104 and_swiglu_kernel_forward_inferenceL196:_l2_swizzle(pid, S, hidden_dim, ...)_swiglu_kernel_backward_dGUL263:_l2_swizzle(pid, S, hidden_dim, ...)_swiglu_kernel_backward_dIL362:_l2_swizzle(pid, S, dim, ...)
The launch grid is computed on the host with Python ints (triton.cdiv(S, BLOCK_M) * triton.cdiv(hidden_dim, BLOCK_N)), so the launch itself goes through and nothing reports an error; the output just has missing and duplicated tiles.
To be upfront: this only bites when one of the dimensions is within BLOCK_M/BLOCK_N of 2^31 (in the runs below the other one is 1024), and the outputs [B, S, hidden_dim] are then terabytes, so the full kernels cannot be run at these shapes on current GPUs. It is the same 32-bit index arithmetic as #804, #1482, #1497 and #1570, where offsets were widened to int64, so it seemed worth widening here too.
Reproduce
The kernels themselves can't be launched at these sizes, so this runs _l2_swizzle (copied unchanged) for the first and last 16 program ids of the grid the wrapper would launch, and compares with the same arithmetic on Python ints:
import torch
import triton
import triton.language as tl
@triton.jit
def _l2_swizzle(pid, M, N, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, GROUP_SIZE_M: tl.constexpr):
# copied from src/liger_kernel/ops/mlp.py
num_pid_m = tl.cdiv(M, BLOCK_M)
num_pid_n = tl.cdiv(N, BLOCK_N)
num_pid_in_group = GROUP_SIZE_M * num_pid_n
group_id = pid // num_pid_in_group
base_pid_m = group_id * GROUP_SIZE_M
effective_group_size_m = tl.minimum(num_pid_m - base_pid_m, GROUP_SIZE_M)
pid_in_group = pid % num_pid_in_group
pid_m = pid_in_group % effective_group_size_m + base_pid_m
pid_n = pid_in_group // effective_group_size_m
return pid_m, pid_n
@triton.jit
def swizzle_kernel(pids, out, S, hidden_dim, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, GROUP_SIZE_M: tl.constexpr):
i = tl.program_id(0)
pid_m, pid_n = _l2_swizzle(tl.load(pids + i), S, hidden_dim, BLOCK_M, BLOCK_N, GROUP_SIZE_M)
tl.store(out + 2 * i, pid_m)
tl.store(out + 2 * i + 1, pid_n)
def reference(pid, M, N, BLOCK_M, BLOCK_N, GROUP_SIZE_M): # the same, on Python ints
num_pid_m, num_pid_n = -(-M // BLOCK_M), -(-N // BLOCK_N)
num_pid_in_group = GROUP_SIZE_M * num_pid_n
base_pid_m = pid // num_pid_in_group * GROUP_SIZE_M
effective_group_size_m = min(num_pid_m - base_pid_m, GROUP_SIZE_M)
pid_in_group = pid % num_pid_in_group
return pid_in_group % effective_group_size_m + base_pid_m, pid_in_group // effective_group_size_m
BLOCK_M, BLOCK_N, GROUP_SIZE_M = 64, 128, 8
for S, hidden_dim in [(2**31 - 1, 1024), (1024, 2**31 - 1), (2**31 - BLOCK_M, 1024), (1024, 2**31 - BLOCK_N)]:
grid = triton.cdiv(S, BLOCK_M) * triton.cdiv(hidden_dim, BLOCK_N) # as the wrapper launches it
pids = list(range(16)) + list(range(grid - 16, grid))
out = torch.empty(2 * len(pids), device="cuda", dtype=torch.int32)
swizzle_kernel[(len(pids),)](torch.tensor(pids, device="cuda", dtype=torch.int32), out, S, hidden_dim,
BLOCK_M=BLOCK_M, BLOCK_N=BLOCK_N, GROUP_SIZE_M=GROUP_SIZE_M)
got = [tuple(t) for t in out.view(-1, 2).tolist()]
wrong = [(p, g, reference(p, S, hidden_dim, BLOCK_M, BLOCK_N, GROUP_SIZE_M))
for p, g in zip(pids, got) if g != reference(p, S, hidden_dim, BLOCK_M, BLOCK_N, GROUP_SIZE_M)]
print(f"S={S}, hidden_dim={hidden_dim}: {len(wrong)}/{len(pids)} tiles wrong", wrong[:2])
S=2147483647, hidden_dim=1024: 24/32 tiles wrong [(8, (8, 0), (0, 1)), (9, (9, 0), (1, 1))]
S=1024, hidden_dim=2147483647: 16/32 tiles wrong [(268435440, (-16, 0), (8, 16777214)), (268435441, (-15, 0), (9, 16777214))]
S=2147483584, hidden_dim=1024: 0/32 tiles wrong []
S=1024, hidden_dim=2147483520: 0/32 tiles wrong []
Program 8 should compute tile (0, 1) but computes (8, 0), which is program 64's tile; near the end of the second grid pid_m comes out negative. With S = 2**31 - BLOCK_M and hidden_dim = 2**31 - BLOCK_N (the largest values whose cdiv still fits in int32) every tile is right.
The fix is to count the tiles in int64 and narrow the result again (pid_m < cdiv(S, BLOCK_M), so pid_m * BLOCK_M and the descriptor offsets stay int32):
@triton.jit
def _l2_swizzle(pid, M, N, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, GROUP_SIZE_M: tl.constexpr):
- num_pid_m = tl.cdiv(M, BLOCK_M)
- num_pid_n = tl.cdiv(N, BLOCK_N)
+ # int64: M + BLOCK_M - 1 overflows int32 once M > 2**31 - BLOCK_M
+ num_pid_m = tl.cdiv(M.to(tl.int64), BLOCK_M)
+ num_pid_n = tl.cdiv(N.to(tl.int64), BLOCK_N)
num_pid_in_group = GROUP_SIZE_M * num_pid_n
group_id = pid // num_pid_in_group
base_pid_m = group_id * GROUP_SIZE_M
@@
pid_in_group = pid % num_pid_in_group
pid_m = pid_in_group % effective_group_size_m + base_pid_m
pid_n = pid_in_group // effective_group_size_m
- return pid_m, pid_n
+ return pid_m.to(tl.int32), pid_n.to(tl.int32)
With this change the script above prints 0/32 tiles wrong for all four shapes. The patch is attached.
Versions
torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Liger-Kernel main @ 781a8ae (_l2_swizzle is the same at 488acbd).
- Lingua principale
- Python
- Stelle
- 6.7k
- Fork
- 668
- Merge medio
- 4g 12h
- PR unite (30g)
- 27
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di linkedin/Liger-Kernel
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsForse già presa @nazek52 l’ha presa 1 giorno fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
linkedin/Liger-Kernel#1647 ·
I maintainer di solito rispondono entro 3 giorni
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Forse già presa @0xDexe l’ha presa 8 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
linkedin/Liger-Kernel#1570 · 3 commenti ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 68/100
linkedin/Liger-Kernel#1661 ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 55/100
linkedin/Liger-Kernel#1660 ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 45/100
linkedin/Liger-Kernel#1659 ·
I maintainer di solito rispondono entro 3 giorni
Tutte le issue di linkedin/Liger-Kernel
Issue simili
-
Internal ErrorApertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
frappe/frappe_docker#2071 ·
I maintainer di solito rispondono entro 1 giorno
-
bug python
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
traceloop/openllmetry#4717 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
The-Pocket/PocketFlow#161 ·
-
[Bug]: Webapp workspace fails to render in Safari — "Can't find variable: requestIdleCallback"Apertacomp/desktop P2 type/bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
NousResearch/hermes-agent#157702 ·
I maintainer di solito rispondono entro 1 giorno
-
scibench_physics / scibench_chemistry: a correct answer scores 0 when the unit has a power of tenForse già presa @BahaaMattar l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
EvolvingLMMs-Lab/lmms-eval#1699 ·
I maintainer di solito rispondono entro 1 giorno