[BUG]: tileiras SIGSEGV compiling dynamic MMA loops around a 128x64 panel
还没有人认领这个 Issue。
评估
调研方向
先从 issue 中独立的 Python 复现程序和生成的 TileIR 开始,然后调用日志中在 O1 和 O2 下显示的 tileiras 命令。当精简后的 dynamic-MMA 内核不再导致 SIGSEGV,并且不支持的组合改为产生编译器诊断时,修复就完成了;使用全新的缓存验证两个优化级别。
由索引模型根据 Issue 内容生成。
描述
cuTile Python version
1.5.0. The same reproducer also fails with 1.4.0.
CUDA Toolkit version
13.3 (tileiras V13.3.36)
Which installation method does this occur on?
Pip
Describe the bug
tileiras terminates with SIGSEGV while compiling the standalone kernel below
at optimization level 1 or 2. The reduced trigger combines two runtime TF32 MMA
loops, a 128x32 panel transformation, an intervening store, and a runtime
conditional. Compilation fails before the deliberately small tensors execute.
I expected the program to compile. If its live tile/resource combination is not
supported, I expected a compiler diagnostic rather than a native process crash.
This was reduced from a fused 64-column Cholesky superpanel. The original used
two float32[1,32768,32768] operands and a large grid to reach the specialization;
the compiler-only example needs two float32[1,4,4] operands and one program.
Minimum reproducible example
import os
import torch
import cuda.tile as ct
ConstInt = ct.Constant[int]
OPT_LEVEL = int(os.environ.get("CUTILE_OPT_LEVEL", "2"))
@ct.kernel(opt_level=OPT_LEVEL)
def _left_superpanel(a, out, super_step, block: ConstInt, rows_per_tile: ConstInt):
batch_id = 0
panel_width = 2 * block
row_block = ct.bid(0)
work = ct.load(
a,
(batch_id, row_block, 0),
shape=(1, rows_per_tile, panel_width),
padding_mode=ct.PaddingMode.ZERO,
).reshape((rows_per_tile, panel_width))
cross_panel = ct.load(
a,
(batch_id, 0, 0),
shape=(1, block, block),
padding_mode=ct.PaddingMode.ZERO,
).reshape((block, block))
cols = ct.arange(block, dtype=ct.int32)[None, :]
right_block = ct.load(
out,
(batch_id, 0, 0),
shape=(1, panel_width, panel_width),
padding_mode=ct.PaddingMode.ZERO,
).reshape((panel_width, panel_width))
for prior_super in range(super_step):
left_block = ct.load(
out,
(batch_id, 0, prior_super),
shape=(1, rows_per_tile, panel_width),
padding_mode=ct.PaddingMode.ZERO,
).reshape((rows_per_tile, panel_width))
left = left_block.astype(ct.tfloat32)
right = right_block.astype(ct.tfloat32)
work = ct.mma(left, right, work)
first_panel = ct.extract(work, (0, 0), shape=(rows_per_tile, block))
second_panel = ct.extract(work, (0, 1), shape=(rows_per_tile, block))
second_panel = ct.mma(
first_panel.astype(ct.tfloat32),
cross_panel.astype(ct.tfloat32),
second_panel,
)
for p in range(block):
solved = ct.extract(second_panel, (0, p), shape=(rows_per_tile, 1))
second_panel = ct.where(cols == p, solved, second_panel)
ct.store(
out,
(batch_id, row_block, 0),
second_panel.reshape((1, rows_per_tile, block)),
)
if super_step:
next_diagonal = cross_panel
for prior_super in range(super_step):
factor = ct.load(
out,
(batch_id, 0, 0),
shape=(1, block, panel_width),
padding_mode=ct.PaddingMode.ZERO,
).reshape((block, panel_width))
next_diagonal = ct.mma(
factor.astype(ct.tfloat32),
factor.transpose(0, 1).astype(ct.tfloat32),
next_diagonal,
)
ct.store(
out,
(batch_id, 0, 0),
next_diagonal.reshape((1, block, block)),
)
def main():
a = torch.empty((1, 4, 4), device="cuda", dtype=torch.float32)
out = torch.empty_like(a)
ct.launch(
torch.cuda.current_stream(),
(1,),
_left_superpanel,
(a, out, 0, 32, 128),
)
torch.cuda.synchronize()
if __name__ == "__main__":
main()
Run at either original optimization level in a fresh compiler cache. Crash dumps
are disabled only to avoid the separate masking problem in #92.
run=$(mktemp -d)
CUDA_TILE_CACHE_DIR=off \
CUDA_TILE_TEMP_DIR="$run" \
CUDA_TILE_ENABLE_CRASH_DUMP=0 \
CUTILE_OPT_LEVEL=2 \
python repro.py
Set CUTILE_OPT_LEVEL=1 for the other original configuration.
Relevant log output
Both optimization levels report:
subprocess.CalledProcessError: Command '['/usr/local/cuda/bin/tileiras',
'/tmp/.../kernel....bytecode', '-o',
'/tmp/.../kernel....cubin', '--gpu-name', 'sm_120',
'-O2', '--lineinfo']' died with <Signals.SIGSEGV: 11>.
cuda.tile._exception.TileCompilerExecutionError: Return code -11
Unknown location
For the O1 run, the compiler command contains -O1 and has the same result.
Directly invoking tileiras on the emitted input exits 139 at both O1 and O2 and
emits no cubin. Four independent cold processes, two at each level,
reproduced. The same script still fails at O2 when importing cuTile Python 1.5.0.
Environment
OS: Ubuntu 22.04.5 LTS, Linux 6.8.0-90-generic x86_64
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0
Driver: 580.126.09
CUDA toolkit: 13.3; nvcc 13.3.33; tileiras V13.3.36
Python: 3.13.14
PyTorch: 2.12.0+cu130 (bundled CUDA runtime 13.0)
cuTile Python: 1.5.0; also reproduced on 1.4.0
CPU: AMD EPYC 9355, 16 vCPUs
Other details
The two original source variants were:
- O2;
- its direct O1 variant, which changed only the kernel decorator.
The two sources differ only by changing the decorator from O2 to O1. Their retained
12,226-byte TileIR files are byte-for-byte identical (SHA-256
162fae627522e4be144e959fd1594d0979f133a4b5677f89523f8511ff08928a),
so they are two optimization settings for one compiler input.
The reduced trigger no longer contains Cholesky initialization, factorization,
triangular solves, block-count logic, input-dependent values, or a multi-program
grid. These changes make the reduced form compile:
- remove either runtime MMA loop;
- remove the 32-iteration panel loop;
- remove the store between the two dynamic regions;
- remove the runtime conditional;
- reduce
rows_per_tilefrom 128 to 64; - reduce
blockfrom 32 to 16.
The minimized bytecode compiles when replayed at O0 without --lineinfo, although the unreduced source also crashed at
O0. This suggests the larger original contains a broader trigger, while the small
example isolates a deterministic O1/O2 boundary.
Contributing Guidelines
- I agree to follow cuTile Python's contributing guidelines
- I searched the open bugs and found no duplicate for this report
- 主要语言
- Python
- 星标
- 2.2k
- 派生
- 155
- PR 合并指标
- 30 天内没有已合并 PR
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
NVIDIA/cutile-python 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 82/100
NVIDIA/cutile-python#105 · 2 条评论 ·
-
bug status: needs-triage
难度 3/5 1-2 天 新手友好度 68/100
NVIDIA/cutile-python#102 ·
-
难度 4/5 3-5 天 新手友好度 68/100
NVIDIA/cutile-python#101 ·
-
bug
难度 4/5 3-5 天 新手友好度 52/100
NVIDIA/cutile-python#96 · 1 条评论 ·
-
bug
NVIDIA/cutile-python#95 · 1 条评论 · 已指派 1 人 ·
查看 NVIDIA/cutile-python 的全部 Issue
相似的 Issue
-
enhancement
难度 2/5 1-3 小时 新手友好度 70/100
canonical/paas-charm#368 · 1 条评论 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
-
tech debt
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 1/5 1 小时以内 新手友好度 90/100
StevenBlack/hosts#3256 ·
-
难度 1/5 1 小时以内 新手友好度 90/100
qualcomm/qai-appbuilder#275 ·