TensorRT 11.4.0.106 leaves half of FP16 GroupNorm output unwritten with a dynamic height
メンテナーはふだん 6 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 35/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- numpy, python
調査の方向性
Start with the self-contained groupnorm_minimal.py reproducer in the issue and run it with TensorRT 11.4.0.106 on a supported NVIDIA GPU. Compare the dynamic and fixed profile results; done means the dynamic profile writes all 32 FP16 values and matches the expected output. The report identifies generated GroupNorm CUDA source as the suspected location, but does not name a repository source file.
索引モデルが issue の本文から書いたものです。
説明
Description
A native TensorRT INormalizationLayer configured for GroupNorm returns success but leaves the second channel of each group unchanged in the output buffer. The reproducer builds one GroupNorm operation with identity scale and zero bias and runs it on 32 synthetic FP16 values (64 bytes).
Environment
| Component | Tested configuration |
|---|---|
| TensorRT | tensorrt-cu13==11.4.0.106 |
| OS | Windows 11 Pro x64, build 26300 |
| GPUs | Four GeForce RTX 3060 cards, two GA104 (10de:2487) and two GA106 (10de:2503), all SM 8.6 |
| NVIDIA driver | 616.56, reporting CUDA Driver API 13.4 |
| Python | 3.13.12 with numpy==2.5.4 and cuda-bindings==13.0.3 |
Reproduction
Save the Python block below as groupnorm_minimal.py. Run these commands in a Python environment with the NVIDIA driver available.
python -m pip install numpy==2.5.4 cuda-bindings==13.0.3 tensorrt-cu13==11.4.0.106
python groupnorm_minimal.py
The network input is [1,16,-1,1] in NCHW order, with eight groups and spatial axes mask 0xC. Both engines receive [1,16,2,1] with identical values. The only profile change is the minimum height.
| Engine | MIN | OPT | MAX |
|---|---|---|---|
| Dynamic | [1,16,1,1] |
[1,16,2,1] |
[1,16,2,1] |
| Fixed control | [1,16,2,1] |
[1,16,2,1] |
[1,16,2,1] |
The input is eight repetitions of [-1, -1, +1, +1]. Each group has two channels, mean zero, and variance one. The expected FP16 output is the input after rounding. The program runs each engine twice and pre-fills the output buffer with 0x0000 and 0x3C3C, respectively. A word is counted as untouched when it retains its corresponding prefill in both runs.
Expected and actual results
Both engines should produce all 32 expected FP16 values regardless of the prefill. With the dynamic profile, the 16 values in zero-based channels 1, 3, 5, 7, 9, 11, 13, and 15 retain the prefill. The fixed profile produces all 32 expected values. The reproducer produced this pattern on all four RTX 3060 cards.
TensorRT 11.4.0.106, CUDA driver API 13040, GPU 0
dynamic: 16/32 words retain both prefills; untouched channels=[1, 3, 5, 7, 9, 11, 13, 15]; matches expected=False
fixed: 0/32 words retain both prefills; untouched channels=[]; matches expected=True
REPRODUCED
Root cause
I captured the CUDA source and PTX during engine construction. The generated GroupNorm kernel family __myl_Gnor_0x94d91cb3b6c44782b0bdbbf351529669 contains a missing pair of parentheses in the input and output channel-index expressions, in both the general and specialized kernel entries. With names simplified, the two forms are
// Generated
(index / 1 * height * width) % channelsPerGroup
// Correct grouping of the divisor
(index / (1 * height * width)) % channelsPerGroup
For this input, height = 2, width = 1, and channelsPerGroup = 2. C++ evaluates the generated expression as (index * 2) % 2, which is always zero. Each group therefore addresses its first channel repeatedly and leaves its second channel unchanged. The corrected expression selects both channels.
I changed only the four instances of the divisor grouping in the generated source in a separate diagnostic build. With the same internal compiler and options, the dynamic engine then wrote all 32 elements and matched the expected FP16 output exactly.
As a separate cross-check, the 12×16 vector-kernel reproducer showed the same missing parentheses. Reassembling its original PTX with ptxas 13.4.59 and 13.4.92 at -O0 and -O3 preserved the same 2,304 untouched values out of 3,072. The incorrect indexing is already present in the CUDA source generated by TensorRT.
Self-contained reproducer
"""Reproduce incomplete FP16 GroupNorm output with one TensorRT layer."""
import argparse
from contextlib import ExitStack
from cuda.bindings import driver as cu
import numpy as np
import tensorrt as trt
SHAPE = (1, 16, 2, 1)
def check(result):
status, *values = result
if status != cu.CUresult.CUDA_SUCCESS:
raise RuntimeError(str(status))
return values[0] if values else None
def run(fixed):
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(0)
x = network.add_input("x", trt.float16, (1, 16, -1, 1))
affine_shape = (1, 16, 1, 1)
scale = np.ones(affine_shape, dtype=np.float16)
bias = np.zeros_like(scale)
s = network.add_constant(affine_shape, scale).get_output(0)
b = network.add_constant(affine_shape, bias).get_output(0)
norm = network.add_normalization_v2(x, s, b, 0b1100)
norm.num_groups = 8
norm.epsilon = 1e-5
y = norm.get_output(0)
y.name = "y"
network.mark_output(y)
config = builder.create_builder_config()
config.max_aux_streams = 0
profile = builder.create_optimization_profile()
profile.set_shape("x", SHAPE if fixed else (1, 16, 1, 1), SHAPE, SHAPE)
config.add_optimization_profile(profile)
plan = builder.build_serialized_network(network, config)
if plan is None:
raise RuntimeError("TensorRT engine build failed")
runtime = trt.Runtime(logger)
engine = runtime.deserialize_cuda_engine(plan)
context = engine.create_execution_context()
if not context.set_input_shape("x", SHAPE) or tuple(context.get_tensor_shape("y")) != SHAPE:
raise RuntimeError("Unexpected input or output shape")
if any(engine.get_tensor_format(name) != trt.TensorFormat.LINEAR
or engine.get_tensor_dtype(name) != trt.float16 for name in ("x", "y")):
raise RuntimeError("Expected linear FP16 I/O")
# Each group has mean 0 and variance 1. The expected FP16 output is the input.
data = np.tile(np.array([-1, -1, 1, 1], dtype=np.float16), 8)
output = np.empty_like(data)
outputs = []
with ExitStack() as buffers:
x_ptr = check(cu.cuMemAlloc(data.nbytes))
buffers.callback(lambda: check(cu.cuMemFree(x_ptr)))
y_ptr = check(cu.cuMemAlloc(output.nbytes))
buffers.callback(lambda: check(cu.cuMemFree(y_ptr)))
addresses = {"x": x_ptr, "y": y_ptr}
bindings = [int(addresses[engine.get_tensor_name(i)]) for i in range(engine.num_io_tensors)]
check(cu.cuMemcpyHtoD(x_ptr, data.ctypes.data, data.nbytes))
for fill in (0, 0x3C):
check(cu.cuMemsetD8(y_ptr, fill, output.nbytes))
check(cu.cuCtxSynchronize())
if not context.execute_v2(bindings):
raise RuntimeError("TensorRT execution failed")
check(cu.cuMemcpyDtoH(output.ctypes.data, y_ptr, output.nbytes))
outputs.append(output.copy())
untouched = (outputs[0].view(np.uint16) == 0) & (outputs[1].view(np.uint16) == 0x3C3C)
channels = np.flatnonzero(untouched.reshape(16, 2).all(axis=1)).tolist()
matches = all(np.array_equal(result, data) for result in outputs)
print(f'{"fixed" if fixed else "dynamic"}: {untouched.sum()}/32 words retain both prefills; '
f"untouched channels={channels}; matches expected={matches}")
return int(untouched.sum()), matches
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--device", type=int, default=0)
args = parser.parse_args()
check(cu.cuInit(0))
gpu = check(cu.cuDeviceGet(args.device))
primary = check(cu.cuDevicePrimaryCtxRetain(gpu))
try:
check(cu.cuCtxSetCurrent(primary))
print(f"TensorRT {trt.__version__}, CUDA driver API {check(cu.cuDriverGetVersion())}, GPU {args.device}")
untouched, matches = run(fixed=False)
if run(fixed=True) != (0, True):
raise RuntimeError("Fixed-profile control failed")
print("REPRODUCED" if untouched and not matches else "NOT REPRODUCED")
finally:
check(cu.cuDevicePrimaryCtxRelease(gpu))
if __name__ == "__main__":
main()
- 主要言語
- C++
- スター
- 13.5k
- フォーク
- 2.4k
- 平均マージ
- 6日 21時間
- マージ済み PR(30日)
- 3
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
NVIDIA/TensorRT のほかの issue
-
Module:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
メンテナーはふだん 6 日以内に返信
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectly対応中かも @0z5a が 16 日前に担当しました。 オープンModule:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 6 日以内に返信
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerance対応中かも @0z5a が 16 日前に担当しました。 オープンModule:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 6 日以内に返信
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop node対応中かも @gabortar が 39 日前に担当しました。 オープンModule:GraphSurgeon
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 6 日以内に返信
-
Polygraphy parse_meta accepts a shape with a missing bracket対応中かも @SashaMIT が 51 日前に担当しました。 オープンModule:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 86/100
メンテナーはふだん 6 日以内に返信
NVIDIA/TensorRT の issue をすべて見る
似ている issue
-
難易度 1/5 1時間未満 初心者へのやさしさ 91/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 69/100
maplibre/maplibre-native#5072 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 79/100
tenstorrent/tt-umd#4017 ·
メンテナーはふだん 3 日以内に返信
-
difficulty:easy func:gui prio:low type:bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
NatronGitHub/Natron#1538 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
hyprwm/hyprsunset#103 · コメント 1 件 ·