Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[BUG]: tileiras SIGSEGV compiling dynamic MMA loops around a 128x64 panel

Aberta
#97 1 comentário 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
45/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Pouca atividade
Stack de tecnologia
python
Domínio
compilers

Direção de pesquisa

Comece com o reproducer Python autônomo na issue e o TileIR emitido e, em seguida, invoque o comando tileiras mostrado nos logs em O1 e O2. A correção estará completa quando o kernel dynamic-MMA reduzido não causar mais SIGSEGV e as combinações não suportadas produzirem um diagnóstico do compilador; verifique ambos os níveis de otimização com um cache novo.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug
cuTile Python version

1.5.0. The same reproducer also fails with 1.4.0.

CUDA Toolkit version

13.3 (tileiras V13.3.36)

Which installation method does this occur on?

Pip

Describe the bug

tileiras terminates with SIGSEGV while compiling the standalone kernel below
at optimization level 1 or 2. The reduced trigger combines two runtime TF32 MMA
loops, a 128x32 panel transformation, an intervening store, and a runtime
conditional. Compilation fails before the deliberately small tensors execute.

I expected the program to compile. If its live tile/resource combination is not
supported, I expected a compiler diagnostic rather than a native process crash.

This was reduced from a fused 64-column Cholesky superpanel. The original used
two float32[1,32768,32768] operands and a large grid to reach the specialization;
the compiler-only example needs two float32[1,4,4] operands and one program.

Minimum reproducible example
import os
import torch
import cuda.tile as ct


ConstInt = ct.Constant[int]
OPT_LEVEL = int(os.environ.get("CUTILE_OPT_LEVEL", "2"))


@ct.kernel(opt_level=OPT_LEVEL)
def _left_superpanel(a, out, super_step, block: ConstInt, rows_per_tile: ConstInt):
    batch_id = 0
    panel_width = 2 * block
    row_block = ct.bid(0)
    work = ct.load(
        a,
        (batch_id, row_block, 0),
        shape=(1, rows_per_tile, panel_width),
        padding_mode=ct.PaddingMode.ZERO,
    ).reshape((rows_per_tile, panel_width))
    cross_panel = ct.load(
        a,
        (batch_id, 0, 0),
        shape=(1, block, block),
        padding_mode=ct.PaddingMode.ZERO,
    ).reshape((block, block))
    cols = ct.arange(block, dtype=ct.int32)[None, :]
    right_block = ct.load(
        out,
        (batch_id, 0, 0),
        shape=(1, panel_width, panel_width),
        padding_mode=ct.PaddingMode.ZERO,
    ).reshape((panel_width, panel_width))

    for prior_super in range(super_step):
        left_block = ct.load(
            out,
            (batch_id, 0, prior_super),
            shape=(1, rows_per_tile, panel_width),
            padding_mode=ct.PaddingMode.ZERO,
        ).reshape((rows_per_tile, panel_width))
        left = left_block.astype(ct.tfloat32)
        right = right_block.astype(ct.tfloat32)
        work = ct.mma(left, right, work)

    first_panel = ct.extract(work, (0, 0), shape=(rows_per_tile, block))
    second_panel = ct.extract(work, (0, 1), shape=(rows_per_tile, block))
    second_panel = ct.mma(
        first_panel.astype(ct.tfloat32),
        cross_panel.astype(ct.tfloat32),
        second_panel,
    )

    for p in range(block):
        solved = ct.extract(second_panel, (0, p), shape=(rows_per_tile, 1))
        second_panel = ct.where(cols == p, solved, second_panel)

    ct.store(
        out,
        (batch_id, row_block, 0),
        second_panel.reshape((1, rows_per_tile, block)),
    )

    if super_step:
        next_diagonal = cross_panel
        for prior_super in range(super_step):
            factor = ct.load(
                out,
                (batch_id, 0, 0),
                shape=(1, block, panel_width),
                padding_mode=ct.PaddingMode.ZERO,
            ).reshape((block, panel_width))
            next_diagonal = ct.mma(
                factor.astype(ct.tfloat32),
                factor.transpose(0, 1).astype(ct.tfloat32),
                next_diagonal,
            )
        ct.store(
            out,
            (batch_id, 0, 0),
            next_diagonal.reshape((1, block, block)),
        )


def main():
    a = torch.empty((1, 4, 4), device="cuda", dtype=torch.float32)
    out = torch.empty_like(a)
    ct.launch(
        torch.cuda.current_stream(),
        (1,),
        _left_superpanel,
        (a, out, 0, 32, 128),
    )
    torch.cuda.synchronize()


if __name__ == "__main__":
    main()

Run at either original optimization level in a fresh compiler cache. Crash dumps
are disabled only to avoid the separate masking problem in #92.

run=$(mktemp -d)
CUDA_TILE_CACHE_DIR=off \
CUDA_TILE_TEMP_DIR="$run" \
CUDA_TILE_ENABLE_CRASH_DUMP=0 \
CUTILE_OPT_LEVEL=2 \
python repro.py

Set CUTILE_OPT_LEVEL=1 for the other original configuration.

Relevant log output

Both optimization levels report:

subprocess.CalledProcessError: Command '['/usr/local/cuda/bin/tileiras',
  '/tmp/.../kernel....bytecode', '-o',
  '/tmp/.../kernel....cubin', '--gpu-name', 'sm_120',
  '-O2', '--lineinfo']' died with <Signals.SIGSEGV: 11>.

cuda.tile._exception.TileCompilerExecutionError: Return code -11
Unknown location

For the O1 run, the compiler command contains -O1 and has the same result.
Directly invoking tileiras on the emitted input exits 139 at both O1 and O2 and
emits no cubin. Four independent cold processes, two at each level,
reproduced. The same script still fails at O2 when importing cuTile Python 1.5.0.

Environment
OS: Ubuntu 22.04.5 LTS, Linux 6.8.0-90-generic x86_64
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0
Driver: 580.126.09
CUDA toolkit: 13.3; nvcc 13.3.33; tileiras V13.3.36
Python: 3.13.14
PyTorch: 2.12.0+cu130 (bundled CUDA runtime 13.0)
cuTile Python: 1.5.0; also reproduced on 1.4.0
CPU: AMD EPYC 9355, 16 vCPUs
Other details

The two original source variants were:

  • O2;
  • its direct O1 variant, which changed only the kernel decorator.

The two sources differ only by changing the decorator from O2 to O1. Their retained
12,226-byte TileIR files are byte-for-byte identical (SHA-256
162fae627522e4be144e959fd1594d0979f133a4b5677f89523f8511ff08928a),
so they are two optimization settings for one compiler input.

The reduced trigger no longer contains Cholesky initialization, factorization,
triangular solves, block-count logic, input-dependent values, or a multi-program
grid. These changes make the reduced form compile:

  • remove either runtime MMA loop;
  • remove the 32-iteration panel loop;
  • remove the store between the two dynamic regions;
  • remove the runtime conditional;
  • reduce rows_per_tile from 128 to 64;
  • reduce block from 32 to 16.

The minimized bytecode compiles when replayed at O0 without --lineinfo, although the unreduced source also crashed at
O0. This suggests the larger original contains a broader trigger, while the small
example isolates a deterministic O1/O2 boundary.

Contributing Guidelines
  • I agree to follow cuTile Python's contributing guidelines
  • I searched the open bugs and found no duplicate for this report
Linguagem predominante
Python
Estrelas
2.2k
Forks
155
Métricas de merge de PRs
Nenhum PR com merge em 30d

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de NVIDIA/cutile-python

Todas as issues de NVIDIA/cutile-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.