Incorrect output failure of TensorRT 11.1 and 11.2 when running dynamic If → Transpose → Resize on GPU GTX 1660 SUPER and A10G
维护者通常 6 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 活跃
- 技术栈
- python
调研方向
将独立复现程序保存为 reproduce_trt_conditional_transpose.py,并针对 ONNX parser 和 --api 路径执行其 build 和 run 操作。在运行时 shape [4,1,19,31] 下,将 TensorRT 输出与 ONNX Runtime 的 CPU 参考结果进行比较,并使用 transpose-before 情况作为对照。完成的标准是,失败情况满足脚本的 np.testing.assert_allclose 检查,而不是产生报告中的大误差。
由索引模型根据 Issue 内容生成。
描述
Description
TensorRT successfully builds a small FP32 If → Transpose → Resize network, but produces substantially incorrect results at a valid dynamic input shape. This reproduces with 11.1.0.106 on A10G and GTX 1660 SUPER, and persists with 11.2.1.2 on GTX 1660 SUPER, through both the ONNX parser and the direct TensorRT network API. A10G has not been tested with 11.2.
The operation is equivalent to:
selected = x + 2 if condition else x + 1
oriented = selected.transpose(0, 1, 3, 2)
y = resize(oriented, size, mode="cubic", align_corners=True, cubic_coeff=-0.75)
At runtime shape [4,1,19,31], the maximum absolute error against ONNX Runtime CPU is 14.6497345. Moving Transpose before If makes the same comparison pass.
Environment
| Cloud reproduction | Local reproduction | |
|---|---|---|
| GPU | NVIDIA A10G, SM86 (AWS g5.2xlarge) | NVIDIA GeForce GTX 1660 SUPER, SM75 |
TensorRT (tensorrt-cu13) |
11.1.0.106 | 11.1.0.106 and 11.2.1.2 (separate environments) |
| NVIDIA driver | 580.178.04 | 610.57.04 |
| CUDA runtime | 13.3.29; runtime API version 13030 | 13.3; toolkit nvcc 13.3.73 |
| Platform | Linux x86_64, Python container | Arch Linux x86_64, bare metal |
| Python | 3.13.15 | 3.13.2 |
| NumPy / ONNX / ONNX Runtime | 2.5.1 / 1.22.0 / 1.27.0 | 2.5.1 / 1.22.0 / 1.27.0 |
Minimal reproducer
Save the following code as reproduce_trt_conditional_transpose.py. It generates synthetic inputs, the ONNX reference and the TensorRT engine. With --api, the TensorRT build uses the network API directly and does not invoke the ONNX parser.
Standalone Python reproducer
"""Standalone synthetic TensorRT If -> Transpose -> Resize reproducer."""
import argparse
import ctypes
import ctypes.util
from pathlib import Path
import numpy as np
import onnx
from onnx import helper as h
from onnx import numpy_helper as nh
def model(batch: int, mode: str, transpose: bool = True, before: bool = False) -> onnx.ModelProto:
vi = h.make_tensor_value_info
branch_shape = [batch, 1, "W", "H"] if transpose and before else [batch, 1, "H", "W"]
branch_input = "ORIENTED" if transpose and before else "X"
yes = h.make_graph([h.make_node("Add", [branch_input, "TWO"], ["YES"])], "then", [], [vi("YES", 1, branch_shape)])
no = h.make_graph([h.make_node("Add", [branch_input, "ONE"], ["NO"])], "else", [], [vi("NO", 1, branch_shape)])
nodes = [h.make_node("If", ["COND"], ["SELECTED"], then_branch=yes, else_branch=no)]
if transpose and before:
nodes.insert(0, h.make_node("Transpose", ["X"], ["ORIENTED"], perm=[0, 1, 3, 2]))
if transpose and not before:
nodes.append(h.make_node("Transpose", ["SELECTED"], ["ORIENTED"], perm=[0, 1, 3, 2]))
nodes.extend(
[
h.make_node("Cast", ["SIZE"], ["SIZE64"], to=7),
h.make_node("Concat", ["NC", "SIZE64"], ["TARGET"], axis=0),
h.make_node(
"Resize",
["ORIENTED" if transpose and not before else "SELECTED", "", "", "TARGET"],
["Y"],
mode=mode,
coordinate_transformation_mode="align_corners",
cubic_coeff_a=-0.75,
),
]
)
weights = [
nh.from_array(np.full((1, 1, 1, 1), 2.0, np.float32), "TWO"),
nh.from_array(np.ones((1, 1, 1, 1), np.float32), "ONE"),
nh.from_array(np.array([batch, 1], np.int64), "NC"),
]
result = h.make_model(
h.make_graph(
nodes,
"conditional-transpose",
[vi("X", 1, [batch, 1, "H", "W"]), vi("COND", 9, []), vi("SIZE", 6, [2])],
[vi("Y", 1, [batch, 1, "OH", "OW"])],
initializer=weights,
),
opset_imports=[h.make_opsetid("", 18)],
ir_version=10,
)
onnx.checker.check_model(result, full_check=True)
return result
def build(args: argparse.Namespace) -> None:
import tensorrt as trt # noqa: PLC0415 - optional GPU dependency
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(0)
if args.api:
x = network.add_input("X", trt.float32, (args.batch, 1, -1, -1))
cond = network.add_input("COND", trt.bool, ())
size = network.add_input("SIZE", trt.int32, (2,))
if not args.no_transpose and args.before:
orient = network.add_shuffle(x)
orient.second_transpose = (0, 1, 3, 2)
x = orient.get_output(0)
branch = network.add_if_conditional()
branch.set_condition(cond)
inside = branch.add_input(x).get_output(0)
one = np.ones((1, 1, 1, 1), np.float32)
two = one * 2
yes = network.add_elementwise(
inside, network.add_constant(two.shape, two).get_output(0), trt.ElementWiseOperation.SUM
)
no = network.add_elementwise(
inside, network.add_constant(one.shape, one).get_output(0), trt.ElementWiseOperation.SUM
)
value = branch.add_output(yes.get_output(0), no.get_output(0)).get_output(0)
if not args.no_transpose and not args.before:
orient = network.add_shuffle(value)
orient.second_transpose = (0, 1, 3, 2)
value = orient.get_output(0)
nc = np.array([args.batch, 1], np.int32)
sizes = network.add_concatenation([network.add_constant(nc.shape, nc).get_output(0), size])
sizes.axis = 0
resize = network.add_resize(value)
resize.resize_mode = getattr(trt.InterpolationMode, args.interpolation.upper())
resize.coordinate_transformation = trt.ResizeCoordinateTransformation.ALIGN_CORNERS
resize.cubic_coeff = -0.75
resize.set_input(1, sizes.get_output(0))
result = resize.get_output(0)
result.name = "Y"
network.mark_output(result)
else:
parser = trt.OnnxParser(network, logger)
assert parser.parse(args.onnx.read_bytes())
config = builder.create_builder_config()
config.clear_flag(trt.BuilderFlag.TF32)
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 512 << 20)
config.profiling_verbosity = trt.ProfilingVerbosity.DETAILED
profile = builder.create_optimization_profile()
profile.set_shape(
"X",
(args.batch, 1, 1, 1),
(args.batch, 1, args.opt_height, args.opt_width),
(args.batch, 1, 1536, 3072),
)
profile.set_shape_input("SIZE", (1, 1), (512, 512), (3072, 3072))
config.add_optimization_profile(profile)
plan = builder.build_serialized_network(network, config)
assert plan is not None
args.plan.write_bytes(bytes(plan))
def run(a: argparse.Namespace) -> None: # noqa: PLR0915 - keep raw CUDA execution explicit
import onnxruntime as ort # noqa: PLC0415 - only required by the runner
import tensorrt as trt # noqa: PLC0415 - optional GPU dependency
cuda = ctypes.CDLL(a.cudart or ctypes.util.find_library("cudart") or "libcudart.so")
for name, args in [
("cudaSetDevice", [ctypes.c_int]),
("cudaMalloc", [ctypes.POINTER(ctypes.c_void_p), ctypes.c_size_t]),
("cudaFree", [ctypes.c_void_p]),
("cudaMemcpy", [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int]),
("cudaStreamCreate", [ctypes.POINTER(ctypes.c_void_p)]),
("cudaStreamSynchronize", [ctypes.c_void_p]),
("cudaDeviceSynchronize", []),
("cudaStreamDestroy", [ctypes.c_void_p]),
]:
fn = getattr(cuda, name)
fn.argtypes = args
fn.restype = ctypes.c_int
def call(name: str, *args: object) -> None:
code = getattr(cuda, name)(*args)
if code:
raise RuntimeError(f"{name}: CUDA {code}")
def allocate(array: np.ndarray) -> ctypes.c_void_p:
ptr = ctypes.c_void_p()
call("cudaMalloc", ctypes.byref(ptr), array.nbytes)
call("cudaMemcpy", ptr, ctypes.c_void_p(array.ctypes.data), array.nbytes, 1)
return ptr
call("cudaSetDevice", 0)
runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))
engine = runtime.deserialize_cuda_engine(a.plan.read_bytes())
context = engine.create_execution_context()
stream = ctypes.c_void_p()
if not a.default_stream:
call("cudaStreamCreate", ctypes.byref(stream))
handle = stream.value or 0
assert context.set_optimization_profile_async(0, handle)
call("cudaStreamSynchronize", stream)
opt = ort.SessionOptions()
opt.intra_op_num_threads = 1
ref = ort.InferenceSession(str(a.onnx), sess_options=opt, providers=["CPUExecutionProvider"])
x = (np.arange(a.batch * a.height * a.width, dtype=np.float32).reshape(a.batch, 1, a.height, a.width) % 113) * 0.1
for condition in (True, False, True):
feed = {"X": x, "COND": np.array(condition, np.bool_), "SIZE": np.array([19, 33], np.int32)}
expected = ref.run(None, feed)[0]
allocs = {}
for name, value in feed.items():
ptr = ctypes.c_void_p(value.ctypes.data)
if engine.get_tensor_location(name) == trt.TensorLocation.DEVICE:
ptr = allocate(value)
allocs[name] = ptr
assert context.set_input_shape(name, tuple(value.shape))
assert context.set_tensor_address(name, ptr.value)
assert not context.infer_shapes()
actual = np.full(tuple(context.get_tensor_shape("Y")), np.nan, np.float32)
out = allocate(actual)
allocs["Y"] = out
assert context.set_tensor_address("Y", out.value)
call("cudaDeviceSynchronize")
print("execute raw", a.batch, a.api, condition, flush=True)
assert context.execute_async_v3(handle)
call("cudaStreamSynchronize", stream)
call("cudaMemcpy", ctypes.c_void_p(actual.ctypes.data), out, actual.nbytes, 2)
print(
"actual",
actual.min(),
actual.max(),
"expected",
expected.min(),
expected.max(),
"max_error",
np.max(abs(actual - expected)),
flush=True,
)
for ptr in allocs.values():
call("cudaFree", ptr)
np.testing.assert_allclose(actual, expected, rtol=1e-5, atol=1e-4)
print("PASS raw", flush=True)
if not a.default_stream:
call("cudaStreamDestroy", stream)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("action", choices=("build", "run"))
parser.add_argument("--batch", type=int, default=4)
parser.add_argument("--height", type=int, default=19)
parser.add_argument("--width", type=int, default=31)
parser.add_argument("--opt-height", type=int, default=1024)
parser.add_argument("--opt-width", type=int, default=1024)
parser.add_argument("--api", action="store_true", help="Bypass the ONNX parser when building TensorRT")
parser.add_argument("--no-transpose", action="store_true")
parser.add_argument("--before", action="store_true", help="Move Transpose before If (control case)")
parser.add_argument("--default-stream", action="store_true")
parser.add_argument("--cudart", help="Path to libcudart.so when unavailable on the library search path")
parser.add_argument("--interpolation", choices=("cubic", "linear"), default="cubic")
parser.add_argument("--directory", type=Path, default=Path("dist/source-prepost/conditional-investigation/repro"))
args = parser.parse_args()
args.directory.mkdir(parents=True, exist_ok=True)
order = "plain" if args.no_transpose else ("transpose-before" if args.before else "transpose")
stem = f"b{args.batch}-{args.interpolation}-{order}"
args.onnx = args.directory / f"{stem}.onnx"
args.plan = args.directory / f"{stem}-{'api' if args.api else 'onnx'}.plan"
if args.action == "build":
onnx.save(model(args.batch, args.interpolation, not args.no_transpose, args.before), args.onnx)
build(args)
else:
run(args)
if __name__ == "__main__":
main()
Steps to reproduce
Install the dependencies in a Python 3.13 environment on a CUDA-capable Linux host:
python -m pip install tensorrt-cu13==11.2.1.2 numpy==2.5.1 \
onnx==1.22.0 onnxruntime==1.27.0 nvidia-cuda-runtime==13.3.29
For the 11.1 comparison, use tensorrt-cu13==11.1.0.106 in a separate environment. Locate the installed CUDA runtime and build/run as separate processes:
TRT_REPRO_CUDART=$(python -c 'import pathlib,sysconfig; print(next(pathlib.Path(sysconfig.get_paths()["purelib"]).rglob("libcudart.so.13")))')
python reproduce_trt_conditional_transpose.py build --api --directory repro
python reproduce_trt_conditional_transpose.py run --api --directory repro \
--cudart "$TRT_REPRO_CUDART"
An existing system libcudart.so.13 path can be supplied instead. The script also supports automatic library lookup when --cudart is omitted.
The default case uses these settings:
| Input / setting | Value |
|---|---|
| X | FP32 [4,1,H,W] |
| COND | Scalar BOOL |
| SIZE | INT32 [2] |
| X spatial profile min / opt / max | 1×1 / 1024×1024 / 1536×3072 |
| SIZE profile min / opt / max | 1×1 / 512×512 / 3072×3072 |
| Runtime X / SIZE | [4,1,19,31] / [19,33] |
| Synthetic X | (arange(4*19*31).reshape(4,1,19,31) % 113) * 0.1, FP32 |
| Resize | Cubic, align corners, coefficient −0.75 |
| TF32 / workspace limit | Disabled / 512 MiB |
Omit --api from both commands to reproduce through the ONNX parser. Use a separate directory when changing profile dimensions, since the filename does not encode the profile.
Expected behavior
TensorRT should match the ONNX Runtime CPU result within rtol=1e-5, atol=1e-4 for this valid input shape. The reference output for the first True condition has range 0.5502561…14.6497345.
Actual behavior
Engine construction succeeds, but the first inference produces range
−0.38888925…3.607245, with maximum absolute error 14.6497345:
execute raw 4 True True
actual -0.38888925 3.607245 expected 0.5502561 14.6497345 max_error 14.6497345
AssertionError: Not equal to tolerance rtol=1e-05, atol=0.0001
The B4 direct-API and ONNX paths produce these same values on both tested GPUs. A second fresh-process execution of the B4 direct-API engine on A10G reproduces the same failure.
Have you tried the latest release?: Yes
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes
- 主要语言
- C++
- 星标
- 13.5k
- 派生
- 2.4k
- 平均合并
- 6 天 21 小时
- 30 天内合并 PR
- 3
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
NVIDIA/TensorRT 的其他 Issue
-
Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 6 天内回复
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectly可能已有人在做 @0z5a 于 16 天前认领。 未关闭Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 6 天内回复
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerance可能已有人在做 @0z5a 于 16 天前认领。 未关闭Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 6 天内回复
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop node可能已有人在做 @gabortar 于 39 天前认领。 未关闭Module:GraphSurgeon
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 6 天内回复
-
Polygraphy parse_meta accepts a shape with a missing bracket可能已有人在做 @SashaMIT 于 51 天前认领。 未关闭Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 86/100
维护者通常 6 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 91/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 69/100
maplibre/maplibre-native#5072 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 79/100
tenstorrent/tt-umd#4017 ·
维护者通常 3 天内回复
-
difficulty:easy func:gui prio:low type:bug
难度 2/5 1-3 小时 新手友好度 72/100
NatronGitHub/Natron#1538 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
hyprwm/hyprsunset#103 · 1 条评论 ·