TensorRT 11.4.0.106 leaves half of FP16 GroupNorm output unwritten with a dynamic height

Abierto
#4,923 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 6 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
numpy, python

Línea de trabajo

Start with the self-contained groupnorm_minimal.py reproducer in the issue and run it with TensorRT 11.4.0.106 on a supported NVIDIA GPU. Compare the dynamic and fixed profile results; done means the dynamic profile writes all 32 FP16 values and matches the expected output. The report identifies generated GroupNorm CUDA source as the suspected location, but does not name a repository source file.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Module:Runtime

Description

A native TensorRT INormalizationLayer configured for GroupNorm returns success but leaves the second channel of each group unchanged in the output buffer. The reproducer builds one GroupNorm operation with identity scale and zero bias and runs it on 32 synthetic FP16 values (64 bytes).

Environment

Component Tested configuration
TensorRT tensorrt-cu13==11.4.0.106
OS Windows 11 Pro x64, build 26300
GPUs Four GeForce RTX 3060 cards, two GA104 (10de:2487) and two GA106 (10de:2503), all SM 8.6
NVIDIA driver 616.56, reporting CUDA Driver API 13.4
Python 3.13.12 with numpy==2.5.4 and cuda-bindings==13.0.3

Reproduction

Save the Python block below as groupnorm_minimal.py. Run these commands in a Python environment with the NVIDIA driver available.

python -m pip install numpy==2.5.4 cuda-bindings==13.0.3 tensorrt-cu13==11.4.0.106
python groupnorm_minimal.py

The network input is [1,16,-1,1] in NCHW order, with eight groups and spatial axes mask 0xC. Both engines receive [1,16,2,1] with identical values. The only profile change is the minimum height.

Engine MIN OPT MAX
Dynamic [1,16,1,1] [1,16,2,1] [1,16,2,1]
Fixed control [1,16,2,1] [1,16,2,1] [1,16,2,1]

The input is eight repetitions of [-1, -1, +1, +1]. Each group has two channels, mean zero, and variance one. The expected FP16 output is the input after rounding. The program runs each engine twice and pre-fills the output buffer with 0x0000 and 0x3C3C, respectively. A word is counted as untouched when it retains its corresponding prefill in both runs.

Expected and actual results

Both engines should produce all 32 expected FP16 values regardless of the prefill. With the dynamic profile, the 16 values in zero-based channels 1, 3, 5, 7, 9, 11, 13, and 15 retain the prefill. The fixed profile produces all 32 expected values. The reproducer produced this pattern on all four RTX 3060 cards.

TensorRT 11.4.0.106, CUDA driver API 13040, GPU 0
dynamic: 16/32 words retain both prefills; untouched channels=[1, 3, 5, 7, 9, 11, 13, 15]; matches expected=False
fixed: 0/32 words retain both prefills; untouched channels=[]; matches expected=True
REPRODUCED

Root cause

I captured the CUDA source and PTX during engine construction. The generated GroupNorm kernel family __myl_Gnor_0x94d91cb3b6c44782b0bdbbf351529669 contains a missing pair of parentheses in the input and output channel-index expressions, in both the general and specialized kernel entries. With names simplified, the two forms are

// Generated
(index / 1 * height * width) % channelsPerGroup

// Correct grouping of the divisor
(index / (1 * height * width)) % channelsPerGroup

For this input, height = 2, width = 1, and channelsPerGroup = 2. C++ evaluates the generated expression as (index * 2) % 2, which is always zero. Each group therefore addresses its first channel repeatedly and leaves its second channel unchanged. The corrected expression selects both channels.

I changed only the four instances of the divisor grouping in the generated source in a separate diagnostic build. With the same internal compiler and options, the dynamic engine then wrote all 32 elements and matched the expected FP16 output exactly.

As a separate cross-check, the 12×16 vector-kernel reproducer showed the same missing parentheses. Reassembling its original PTX with ptxas 13.4.59 and 13.4.92 at -O0 and -O3 preserved the same 2,304 untouched values out of 3,072. The incorrect indexing is already present in the CUDA source generated by TensorRT.

Self-contained reproducer

"""Reproduce incomplete FP16 GroupNorm output with one TensorRT layer."""

import argparse
from contextlib import ExitStack

from cuda.bindings import driver as cu
import numpy as np
import tensorrt as trt

SHAPE = (1, 16, 2, 1)


def check(result):
    status, *values = result
    if status != cu.CUresult.CUDA_SUCCESS:
        raise RuntimeError(str(status))
    return values[0] if values else None


def run(fixed):
    logger = trt.Logger(trt.Logger.WARNING)
    builder = trt.Builder(logger)
    network = builder.create_network(0)
    x = network.add_input("x", trt.float16, (1, 16, -1, 1))
    affine_shape = (1, 16, 1, 1)
    scale = np.ones(affine_shape, dtype=np.float16)
    bias = np.zeros_like(scale)
    s = network.add_constant(affine_shape, scale).get_output(0)
    b = network.add_constant(affine_shape, bias).get_output(0)
    norm = network.add_normalization_v2(x, s, b, 0b1100)
    norm.num_groups = 8
    norm.epsilon = 1e-5
    y = norm.get_output(0)
    y.name = "y"
    network.mark_output(y)

    config = builder.create_builder_config()
    config.max_aux_streams = 0
    profile = builder.create_optimization_profile()
    profile.set_shape("x", SHAPE if fixed else (1, 16, 1, 1), SHAPE, SHAPE)
    config.add_optimization_profile(profile)
    plan = builder.build_serialized_network(network, config)
    if plan is None:
        raise RuntimeError("TensorRT engine build failed")
    runtime = trt.Runtime(logger)
    engine = runtime.deserialize_cuda_engine(plan)
    context = engine.create_execution_context()
    if not context.set_input_shape("x", SHAPE) or tuple(context.get_tensor_shape("y")) != SHAPE:
        raise RuntimeError("Unexpected input or output shape")
    if any(engine.get_tensor_format(name) != trt.TensorFormat.LINEAR
           or engine.get_tensor_dtype(name) != trt.float16 for name in ("x", "y")):
        raise RuntimeError("Expected linear FP16 I/O")

    # Each group has mean 0 and variance 1. The expected FP16 output is the input.
    data = np.tile(np.array([-1, -1, 1, 1], dtype=np.float16), 8)
    output = np.empty_like(data)
    outputs = []
    with ExitStack() as buffers:
        x_ptr = check(cu.cuMemAlloc(data.nbytes))
        buffers.callback(lambda: check(cu.cuMemFree(x_ptr)))
        y_ptr = check(cu.cuMemAlloc(output.nbytes))
        buffers.callback(lambda: check(cu.cuMemFree(y_ptr)))
        addresses = {"x": x_ptr, "y": y_ptr}
        bindings = [int(addresses[engine.get_tensor_name(i)]) for i in range(engine.num_io_tensors)]
        check(cu.cuMemcpyHtoD(x_ptr, data.ctypes.data, data.nbytes))
        for fill in (0, 0x3C):
            check(cu.cuMemsetD8(y_ptr, fill, output.nbytes))
            check(cu.cuCtxSynchronize())
            if not context.execute_v2(bindings):
                raise RuntimeError("TensorRT execution failed")
            check(cu.cuMemcpyDtoH(output.ctypes.data, y_ptr, output.nbytes))
            outputs.append(output.copy())

    untouched = (outputs[0].view(np.uint16) == 0) & (outputs[1].view(np.uint16) == 0x3C3C)
    channels = np.flatnonzero(untouched.reshape(16, 2).all(axis=1)).tolist()
    matches = all(np.array_equal(result, data) for result in outputs)
    print(f'{"fixed" if fixed else "dynamic"}: {untouched.sum()}/32 words retain both prefills; '
          f"untouched channels={channels}; matches expected={matches}")
    return int(untouched.sum()), matches


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--device", type=int, default=0)
    args = parser.parse_args()
    check(cu.cuInit(0))
    gpu = check(cu.cuDeviceGet(args.device))
    primary = check(cu.cuDevicePrimaryCtxRetain(gpu))
    try:
        check(cu.cuCtxSetCurrent(primary))
        print(f"TensorRT {trt.__version__}, CUDA driver API {check(cu.cuDriverGetVersion())}, GPU {args.device}")
        untouched, matches = run(fixed=False)
        if run(fixed=True) != (0, True):
            raise RuntimeError("Fixed-profile control failed")
        print("REPRODUCED" if untouched and not matches else "NOT REPRODUCED")
    finally:
        check(cu.cuDevicePrimaryCtxRelease(gpu))


if __name__ == "__main__":
    main()
Lenguaje dominante
C++
Estrellas
13.5k
Forks
2.4k
Merge medio
6 d 21 h
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/TensorRT

Todos los issues de NVIDIA/TensorRT

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.