# [Bug] `_l2_swizzle` in mlp.py counts tiles in int32 (wrong tile mapping when `S` or `hidden_dim` is within a block of 2^31)

Aperta Adatta ai principianti
#1,658 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
77/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Leggi src/liger_kernel/ops/mlp.py, in particolare _l2_swizzle alla riga 33 e i quattro punti di chiamata del kernel SwiGLU elencati nell’issue. Inizia verificando il calcolo del numero di tile rispetto al riferimento con interi Python dello script di riproduzione, per dimensioni vicine a 2^31. Il lavoro è completato quando lo swizzle mappa ogni ID di programma testato sul tile di riferimento, comprese le quattro forme limite descritte.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

🐛 Describe the bug

_l2_swizzle in mlp.py:33 maps a program id to its output tile (pid_m, pid_n). It starts from

num_pid_m = tl.cdiv(M, BLOCK_M)
num_pid_n = tl.cdiv(N, BLOCK_N)

where M and N are the kernel's int32 shape arguments. tl.cdiv(x, d) is (x + d - 1) // d, so once M > 2**31 - BLOCK_M (or N > 2**31 - BLOCK_N) the sum wraps to a negative number. The group arithmetic below it then hands out the wrong tiles: some tiles are computed twice, others never, and some programs get a negative pid_m, so their descriptor loads and stores fall outside the tensor.

The SwiGLU MLP kernels all go through it:

The launch grid is computed on the host with Python ints (triton.cdiv(S, BLOCK_M) * triton.cdiv(hidden_dim, BLOCK_N)), so the launch itself goes through and nothing reports an error; the output just has missing and duplicated tiles.

To be upfront: this only bites when one of the dimensions is within BLOCK_M/BLOCK_N of 2^31 (in the runs below the other one is 1024), and the outputs [B, S, hidden_dim] are then terabytes, so the full kernels cannot be run at these shapes on current GPUs. It is the same 32-bit index arithmetic as #804, #1482, #1497 and #1570, where offsets were widened to int64, so it seemed worth widening here too.

Reproduce

The kernels themselves can't be launched at these sizes, so this runs _l2_swizzle (copied unchanged) for the first and last 16 program ids of the grid the wrapper would launch, and compares with the same arithmetic on Python ints:

import torch
import triton
import triton.language as tl


@triton.jit
def _l2_swizzle(pid, M, N, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, GROUP_SIZE_M: tl.constexpr):
    # copied from src/liger_kernel/ops/mlp.py
    num_pid_m = tl.cdiv(M, BLOCK_M)
    num_pid_n = tl.cdiv(N, BLOCK_N)
    num_pid_in_group = GROUP_SIZE_M * num_pid_n
    group_id = pid // num_pid_in_group
    base_pid_m = group_id * GROUP_SIZE_M
    effective_group_size_m = tl.minimum(num_pid_m - base_pid_m, GROUP_SIZE_M)
    pid_in_group = pid % num_pid_in_group
    pid_m = pid_in_group % effective_group_size_m + base_pid_m
    pid_n = pid_in_group // effective_group_size_m
    return pid_m, pid_n


@triton.jit
def swizzle_kernel(pids, out, S, hidden_dim, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, GROUP_SIZE_M: tl.constexpr):
    i = tl.program_id(0)
    pid_m, pid_n = _l2_swizzle(tl.load(pids + i), S, hidden_dim, BLOCK_M, BLOCK_N, GROUP_SIZE_M)
    tl.store(out + 2 * i, pid_m)
    tl.store(out + 2 * i + 1, pid_n)


def reference(pid, M, N, BLOCK_M, BLOCK_N, GROUP_SIZE_M):  # the same, on Python ints
    num_pid_m, num_pid_n = -(-M // BLOCK_M), -(-N // BLOCK_N)
    num_pid_in_group = GROUP_SIZE_M * num_pid_n
    base_pid_m = pid // num_pid_in_group * GROUP_SIZE_M
    effective_group_size_m = min(num_pid_m - base_pid_m, GROUP_SIZE_M)
    pid_in_group = pid % num_pid_in_group
    return pid_in_group % effective_group_size_m + base_pid_m, pid_in_group // effective_group_size_m


BLOCK_M, BLOCK_N, GROUP_SIZE_M = 64, 128, 8
for S, hidden_dim in [(2**31 - 1, 1024), (1024, 2**31 - 1), (2**31 - BLOCK_M, 1024), (1024, 2**31 - BLOCK_N)]:
    grid = triton.cdiv(S, BLOCK_M) * triton.cdiv(hidden_dim, BLOCK_N)  # as the wrapper launches it
    pids = list(range(16)) + list(range(grid - 16, grid))
    out = torch.empty(2 * len(pids), device="cuda", dtype=torch.int32)
    swizzle_kernel[(len(pids),)](torch.tensor(pids, device="cuda", dtype=torch.int32), out, S, hidden_dim,
                                 BLOCK_M=BLOCK_M, BLOCK_N=BLOCK_N, GROUP_SIZE_M=GROUP_SIZE_M)
    got = [tuple(t) for t in out.view(-1, 2).tolist()]
    wrong = [(p, g, reference(p, S, hidden_dim, BLOCK_M, BLOCK_N, GROUP_SIZE_M))
             for p, g in zip(pids, got) if g != reference(p, S, hidden_dim, BLOCK_M, BLOCK_N, GROUP_SIZE_M)]
    print(f"S={S}, hidden_dim={hidden_dim}: {len(wrong)}/{len(pids)} tiles wrong", wrong[:2])
S=2147483647, hidden_dim=1024: 24/32 tiles wrong [(8, (8, 0), (0, 1)), (9, (9, 0), (1, 1))]
S=1024, hidden_dim=2147483647: 16/32 tiles wrong [(268435440, (-16, 0), (8, 16777214)), (268435441, (-15, 0), (9, 16777214))]
S=2147483584, hidden_dim=1024: 0/32 tiles wrong []
S=1024, hidden_dim=2147483520: 0/32 tiles wrong []

Program 8 should compute tile (0, 1) but computes (8, 0), which is program 64's tile; near the end of the second grid pid_m comes out negative. With S = 2**31 - BLOCK_M and hidden_dim = 2**31 - BLOCK_N (the largest values whose cdiv still fits in int32) every tile is right.

The fix is to count the tiles in int64 and narrow the result again (pid_m < cdiv(S, BLOCK_M), so pid_m * BLOCK_M and the descriptor offsets stay int32):

 @triton.jit
 def _l2_swizzle(pid, M, N, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, GROUP_SIZE_M: tl.constexpr):
-    num_pid_m = tl.cdiv(M, BLOCK_M)
-    num_pid_n = tl.cdiv(N, BLOCK_N)
+    # int64: M + BLOCK_M - 1 overflows int32 once M > 2**31 - BLOCK_M
+    num_pid_m = tl.cdiv(M.to(tl.int64), BLOCK_M)
+    num_pid_n = tl.cdiv(N.to(tl.int64), BLOCK_N)
     num_pid_in_group = GROUP_SIZE_M * num_pid_n
     group_id = pid // num_pid_in_group
     base_pid_m = group_id * GROUP_SIZE_M
@@
     pid_in_group = pid % num_pid_in_group
     pid_m = pid_in_group % effective_group_size_m + base_pid_m
     pid_n = pid_in_group // effective_group_size_m
-    return pid_m, pid_n
+    return pid_m.to(tl.int32), pid_n.to(tl.int32)

With this change the script above prints 0/32 tiles wrong for all four shapes. The patch is attached.

Versions

torch 2.9.1+cu128, triton 3.5.1, CUDA 12.8, NVIDIA GeForce RTX 4060 Ti (8 GB), driver 595.91.07. Liger-Kernel main @ 781a8ae (_l2_swizzle is the same at 488acbd).

Lingua principale
Python
Stelle
6.7k
Fork
668
Merge medio
4g 12h
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di linkedin/Liger-Kernel

Tutte le issue di linkedin/Liger-Kernel

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.