[BUG]: tileiras SIGSEGV when occupancy=2 is requested for a 32-wide fused tile kernel
まだ誰も着手していません。
評価
調査の方向性
まず、示されている cache と crash dump の設定で、新しいプロセス内で repro.py を実行し、次に occupancy=2 を成功するコントロールと比較します。SIGSEGV を返す tileiras の呼び出しを追跡し、列挙されている occupancy、worker-warp、block-size、allocation の各バリアントを使ってコンパイラーの失敗を切り分けます。reproducer がクラッシュしなくなり、失敗する構成を回帰テストでカバーできれば完了です。
索引モデルが issue の本文から書いたものです。
説明
cuTile Python version
1.5.0. The same reproducer also fails with 1.4.0.
CUDA Toolkit version
13.3 (tileiras V13.3.36)
Which installation method does this occur on?
Pip
Describe the bug
tileiras terminates with SIGSEGV when occupancy=2 is requested for the
kernel below. Automatic occupancy and occupancy=1 compile. The crash happens
during compilation, before the intentionally small input can execute.
I expected the occupancy request either to compile, to be treated as a hint that
cannot be met, or to produce a clear resource diagnostic. A scheduling/resource
request should not terminate the native compiler.
This was reduced from two Cholesky _left_superpanel failures. Their source
inputs were float32[1,4096,4096]; the compiler-only reproducer needs one
float32[1,4,4] tensor while preserving the 32x32, 32x64, and 64x64 compile-time
tiles.
Minimum reproducible example
import torch
import cuda.tile as ct
ConstInt = ct.Constant[int]
ZERO = ct.PaddingMode.ZERO
def factor(a, block: ConstInt):
cols = ct.arange(block, dtype=ct.int32)[None, :]
for p in range(block):
pivot = ct.extract(a, (p, p), shape=(1, 1))
column = ct.extract(a, (0, p), shape=(block, 1)) / pivot
outer = column * column.transpose(0, 1)
a = ct.where(cols > p, a - outer, a)
return a
def solve(panel, diagonal, block: ConstInt):
cols = ct.arange(block, dtype=ct.int32)[None, :]
for p in range(block):
pivot = ct.extract(diagonal, (p, p), shape=(1, 1))
solved = ct.extract(panel, (0, p), shape=(block, 1)) / pivot
column = ct.extract(diagonal, (0, p), shape=(block, 1))
panel = ct.where(
cols > p,
panel - solved * column.transpose(0, 1),
panel,
)
return panel
@ct.kernel(opt_level=2, occupancy=2)
def kernel(a, step, block: ConstInt):
rows = block
width = 2 * block
next_step = step + 1
work = ct.load(a, (0, 0, 0), shape=(1, rows, width), padding_mode=ZERO)
work = work.reshape((rows, width))
cross = ct.load(a, (0, 1, 0), shape=(1, block, block), padding_mode=ZERO)
cross = cross.reshape((block, block))
diagonal = ct.load(a, (0, 1, 1), shape=(1, block, block), padding_mode=ZERO)
diagonal = diagonal.reshape((block, block))
for prior in range(step):
left = ct.load(a, (0, 0, prior), shape=(1, rows, width), padding_mode=ZERO)
right = ct.load(a, (0, 0, prior), shape=(1, width, width), padding_mode=ZERO)
left = left.reshape((rows, width))
right = right.reshape((width, width))
work = ct.mma(
left.astype(ct.tfloat32),
(-right.transpose(0, 1)).astype(ct.tfloat32),
work,
)
diagonal = ct.mma(
left.astype(ct.tfloat32),
(-left.transpose(0, 1)).astype(ct.tfloat32),
diagonal,
)
first_panel = ct.extract(work, (0, 0), shape=(rows, block))
second_panel = ct.mma(
first_panel.astype(ct.tfloat32),
(-cross.transpose(0, 1)).astype(ct.tfloat32),
first_panel,
)
second_panel = solve(second_panel, diagonal, block)
ct.store(a, (0, 0, 0), first_panel.reshape((1, rows, block)))
if ct.bid(1) == 0:
ct.store(a, (0, 1, 1), diagonal.reshape((1, block, block)))
if ct.bid(1) == next_step:
next_diagonal = ct.load(
a,
(0, next_step, next_step),
shape=(1, block, block),
padding_mode=ZERO,
).reshape((block, block))
next_diagonal = ct.mma(
second_panel.astype(ct.tfloat32),
(-second_panel.transpose(0, 1)).astype(ct.tfloat32),
next_diagonal,
)
next_diagonal = factor(next_diagonal, block)
ct.store(
a,
(0, next_step, next_step),
next_diagonal.reshape((1, block, block)),
)
a = torch.empty((1, 4, 4), device="cuda", dtype=torch.float32)
ct.launch(torch.cuda.current_stream(), (1, 1), kernel, (a, 0, 32))
Run it in a fresh process and compiler cache. Crash dumps are disabled only to
avoid the separate masking problem in #92.
run=$(mktemp -d)
CUDA_TILE_CACHE_DIR=off \
CUDA_TILE_TEMP_DIR="$run" \
CUDA_TILE_ENABLE_CRASH_DUMP=0 \
python repro.py
Relevant log output
subprocess.CalledProcessError: Command '['/usr/local/cuda/bin/tileiras',
'/tmp/.../kernel....bytecode', '-o',
'/tmp/.../kernel....cubin', '--gpu-name', 'sm_120',
'-O2', '--lineinfo']' died with <Signals.SIGSEGV: 11>.
cuda.tile._exception.TileCompilerExecutionError: Return code -11
Unknown location
The corresponding direct tileiras invocation exits 139 and emits no cubin.
Two fresh 1.4.0 processes produced identical failing bytecode; a fresh 1.5.0
process also produced a failing compiler input.
Environment
OS: Ubuntu 22.04.5 LTS, Linux 6.8.0-90-generic x86_64
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0
Driver: 580.126.09
CUDA toolkit: 13.3; nvcc 13.3.33; tileiras V13.3.36
Python: 3.13.14
PyTorch: 2.12.0+cu130 (bundled CUDA runtime 13.0)
cuTile Python: 1.5.0; also reproduced on 1.4.0
CPU: AMD EPYC 9355, 16 vCPUs
Other details
The two original failing source variants were:
occupancy=2;occupancy=2, num_worker_warps=4.
Both original inputs compiled the same _left_superpanel body and failed at
n=4096. The reduced controls isolate the request:
- automatic occupancy passes;
occupancy=1passes;occupancy=2, num_worker_warps=4has the same crash;occupancy=2, num_worker_warps=8passes at this reduced boundary;occupancy=2withblock=16passes;- shrinking the allocated tensor from 4x4 to 3x3 changes its alignment signature
and passes.
The original no-worker-warp and four-worker-warp compiler inputs differ only by
worker-warp metadata and both exit 139. This makes the occupancy-two request the
common trigger for this compact configuration.
Contributing Guidelines
- I agree to follow cuTile Python's contributing guidelines
- I searched the open bugs and found no duplicate for this report
- 主要言語
- Python
- スター
- 2.2k
- フォーク
- 155
- PR マージ指標
- 30日以内にマージされた PR はありません
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
NVIDIA/cutile-python のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
NVIDIA/cutile-python#105 · コメント 2 件 ·
-
bug status: needs-triage
難易度 3/5 1〜2日 初心者へのやさしさ 68/100
NVIDIA/cutile-python#102 ·
-
難易度 4/5 3〜5日 初心者へのやさしさ 68/100
NVIDIA/cutile-python#101 ·
-
bug
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
NVIDIA/cutile-python#97 · コメント 1 件 ·
-
bug
NVIDIA/cutile-python#95 · コメント 1 件 · 担当者 1 名 ·
NVIDIA/cutile-python の issue をすべて見る
似ている issue
-
enhancement
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
canonical/paas-charm#368 · コメント 1 件 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
-
tech debt
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
-
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
StevenBlack/hosts#3256 ·
-
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
qualcomm/qai-appbuilder#275 ·