Incorrect output failure of TensorRT 11.1 and 11.2 when running dynamic If → Transpose → Resize on GPU GTX 1660 SUPER and A10G
Maintainer antworten meist innerhalb von 6 Tagen
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 35/100
- Issue-Typ
- Bug
- Klarheit
- Muss geklärt werden
- Aktivitätsstatus
- Aktiv
- Tech-Stack
- python
- Bereich
- machine-learning, performance
Rechercherichtung
Speichere den eigenständigen Reproducer als reproduce_trt_conditional_transpose.py und führe seine Build- und Run-Aktionen für die ONNX parser- und --api-Pfade aus. Vergleiche zur Laufzeit bei der Form [4,1,19,31] die TensorRT-Ausgabe mit der CPU-Referenz von ONNX Runtime und verwende dabei den transpose-before-Fall als Kontrolle. Erledigt ist die Aufgabe, wenn der fehlschlagende Fall die np.testing.assert_allclose-Prüfung des Skripts erfüllt, statt den gemeldeten großen Fehler zu erzeugen.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Description
TensorRT successfully builds a small FP32 If → Transpose → Resize network, but produces substantially incorrect results at a valid dynamic input shape. This reproduces with 11.1.0.106 on A10G and GTX 1660 SUPER, and persists with 11.2.1.2 on GTX 1660 SUPER, through both the ONNX parser and the direct TensorRT network API. A10G has not been tested with 11.2.
The operation is equivalent to:
selected = x + 2 if condition else x + 1
oriented = selected.transpose(0, 1, 3, 2)
y = resize(oriented, size, mode="cubic", align_corners=True, cubic_coeff=-0.75)
At runtime shape [4,1,19,31], the maximum absolute error against ONNX Runtime CPU is 14.6497345. Moving Transpose before If makes the same comparison pass.
Environment
| Cloud reproduction | Local reproduction | |
|---|---|---|
| GPU | NVIDIA A10G, SM86 (AWS g5.2xlarge) | NVIDIA GeForce GTX 1660 SUPER, SM75 |
TensorRT (tensorrt-cu13) |
11.1.0.106 | 11.1.0.106 and 11.2.1.2 (separate environments) |
| NVIDIA driver | 580.178.04 | 610.57.04 |
| CUDA runtime | 13.3.29; runtime API version 13030 | 13.3; toolkit nvcc 13.3.73 |
| Platform | Linux x86_64, Python container | Arch Linux x86_64, bare metal |
| Python | 3.13.15 | 3.13.2 |
| NumPy / ONNX / ONNX Runtime | 2.5.1 / 1.22.0 / 1.27.0 | 2.5.1 / 1.22.0 / 1.27.0 |
Minimal reproducer
Save the following code as reproduce_trt_conditional_transpose.py. It generates synthetic inputs, the ONNX reference and the TensorRT engine. With --api, the TensorRT build uses the network API directly and does not invoke the ONNX parser.
Standalone Python reproducer
"""Standalone synthetic TensorRT If -> Transpose -> Resize reproducer."""
import argparse
import ctypes
import ctypes.util
from pathlib import Path
import numpy as np
import onnx
from onnx import helper as h
from onnx import numpy_helper as nh
def model(batch: int, mode: str, transpose: bool = True, before: bool = False) -> onnx.ModelProto:
vi = h.make_tensor_value_info
branch_shape = [batch, 1, "W", "H"] if transpose and before else [batch, 1, "H", "W"]
branch_input = "ORIENTED" if transpose and before else "X"
yes = h.make_graph([h.make_node("Add", [branch_input, "TWO"], ["YES"])], "then", [], [vi("YES", 1, branch_shape)])
no = h.make_graph([h.make_node("Add", [branch_input, "ONE"], ["NO"])], "else", [], [vi("NO", 1, branch_shape)])
nodes = [h.make_node("If", ["COND"], ["SELECTED"], then_branch=yes, else_branch=no)]
if transpose and before:
nodes.insert(0, h.make_node("Transpose", ["X"], ["ORIENTED"], perm=[0, 1, 3, 2]))
if transpose and not before:
nodes.append(h.make_node("Transpose", ["SELECTED"], ["ORIENTED"], perm=[0, 1, 3, 2]))
nodes.extend(
[
h.make_node("Cast", ["SIZE"], ["SIZE64"], to=7),
h.make_node("Concat", ["NC", "SIZE64"], ["TARGET"], axis=0),
h.make_node(
"Resize",
["ORIENTED" if transpose and not before else "SELECTED", "", "", "TARGET"],
["Y"],
mode=mode,
coordinate_transformation_mode="align_corners",
cubic_coeff_a=-0.75,
),
]
)
weights = [
nh.from_array(np.full((1, 1, 1, 1), 2.0, np.float32), "TWO"),
nh.from_array(np.ones((1, 1, 1, 1), np.float32), "ONE"),
nh.from_array(np.array([batch, 1], np.int64), "NC"),
]
result = h.make_model(
h.make_graph(
nodes,
"conditional-transpose",
[vi("X", 1, [batch, 1, "H", "W"]), vi("COND", 9, []), vi("SIZE", 6, [2])],
[vi("Y", 1, [batch, 1, "OH", "OW"])],
initializer=weights,
),
opset_imports=[h.make_opsetid("", 18)],
ir_version=10,
)
onnx.checker.check_model(result, full_check=True)
return result
def build(args: argparse.Namespace) -> None:
import tensorrt as trt # noqa: PLC0415 - optional GPU dependency
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(0)
if args.api:
x = network.add_input("X", trt.float32, (args.batch, 1, -1, -1))
cond = network.add_input("COND", trt.bool, ())
size = network.add_input("SIZE", trt.int32, (2,))
if not args.no_transpose and args.before:
orient = network.add_shuffle(x)
orient.second_transpose = (0, 1, 3, 2)
x = orient.get_output(0)
branch = network.add_if_conditional()
branch.set_condition(cond)
inside = branch.add_input(x).get_output(0)
one = np.ones((1, 1, 1, 1), np.float32)
two = one * 2
yes = network.add_elementwise(
inside, network.add_constant(two.shape, two).get_output(0), trt.ElementWiseOperation.SUM
)
no = network.add_elementwise(
inside, network.add_constant(one.shape, one).get_output(0), trt.ElementWiseOperation.SUM
)
value = branch.add_output(yes.get_output(0), no.get_output(0)).get_output(0)
if not args.no_transpose and not args.before:
orient = network.add_shuffle(value)
orient.second_transpose = (0, 1, 3, 2)
value = orient.get_output(0)
nc = np.array([args.batch, 1], np.int32)
sizes = network.add_concatenation([network.add_constant(nc.shape, nc).get_output(0), size])
sizes.axis = 0
resize = network.add_resize(value)
resize.resize_mode = getattr(trt.InterpolationMode, args.interpolation.upper())
resize.coordinate_transformation = trt.ResizeCoordinateTransformation.ALIGN_CORNERS
resize.cubic_coeff = -0.75
resize.set_input(1, sizes.get_output(0))
result = resize.get_output(0)
result.name = "Y"
network.mark_output(result)
else:
parser = trt.OnnxParser(network, logger)
assert parser.parse(args.onnx.read_bytes())
config = builder.create_builder_config()
config.clear_flag(trt.BuilderFlag.TF32)
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 512 << 20)
config.profiling_verbosity = trt.ProfilingVerbosity.DETAILED
profile = builder.create_optimization_profile()
profile.set_shape(
"X",
(args.batch, 1, 1, 1),
(args.batch, 1, args.opt_height, args.opt_width),
(args.batch, 1, 1536, 3072),
)
profile.set_shape_input("SIZE", (1, 1), (512, 512), (3072, 3072))
config.add_optimization_profile(profile)
plan = builder.build_serialized_network(network, config)
assert plan is not None
args.plan.write_bytes(bytes(plan))
def run(a: argparse.Namespace) -> None: # noqa: PLR0915 - keep raw CUDA execution explicit
import onnxruntime as ort # noqa: PLC0415 - only required by the runner
import tensorrt as trt # noqa: PLC0415 - optional GPU dependency
cuda = ctypes.CDLL(a.cudart or ctypes.util.find_library("cudart") or "libcudart.so")
for name, args in [
("cudaSetDevice", [ctypes.c_int]),
("cudaMalloc", [ctypes.POINTER(ctypes.c_void_p), ctypes.c_size_t]),
("cudaFree", [ctypes.c_void_p]),
("cudaMemcpy", [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int]),
("cudaStreamCreate", [ctypes.POINTER(ctypes.c_void_p)]),
("cudaStreamSynchronize", [ctypes.c_void_p]),
("cudaDeviceSynchronize", []),
("cudaStreamDestroy", [ctypes.c_void_p]),
]:
fn = getattr(cuda, name)
fn.argtypes = args
fn.restype = ctypes.c_int
def call(name: str, *args: object) -> None:
code = getattr(cuda, name)(*args)
if code:
raise RuntimeError(f"{name}: CUDA {code}")
def allocate(array: np.ndarray) -> ctypes.c_void_p:
ptr = ctypes.c_void_p()
call("cudaMalloc", ctypes.byref(ptr), array.nbytes)
call("cudaMemcpy", ptr, ctypes.c_void_p(array.ctypes.data), array.nbytes, 1)
return ptr
call("cudaSetDevice", 0)
runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))
engine = runtime.deserialize_cuda_engine(a.plan.read_bytes())
context = engine.create_execution_context()
stream = ctypes.c_void_p()
if not a.default_stream:
call("cudaStreamCreate", ctypes.byref(stream))
handle = stream.value or 0
assert context.set_optimization_profile_async(0, handle)
call("cudaStreamSynchronize", stream)
opt = ort.SessionOptions()
opt.intra_op_num_threads = 1
ref = ort.InferenceSession(str(a.onnx), sess_options=opt, providers=["CPUExecutionProvider"])
x = (np.arange(a.batch * a.height * a.width, dtype=np.float32).reshape(a.batch, 1, a.height, a.width) % 113) * 0.1
for condition in (True, False, True):
feed = {"X": x, "COND": np.array(condition, np.bool_), "SIZE": np.array([19, 33], np.int32)}
expected = ref.run(None, feed)[0]
allocs = {}
for name, value in feed.items():
ptr = ctypes.c_void_p(value.ctypes.data)
if engine.get_tensor_location(name) == trt.TensorLocation.DEVICE:
ptr = allocate(value)
allocs[name] = ptr
assert context.set_input_shape(name, tuple(value.shape))
assert context.set_tensor_address(name, ptr.value)
assert not context.infer_shapes()
actual = np.full(tuple(context.get_tensor_shape("Y")), np.nan, np.float32)
out = allocate(actual)
allocs["Y"] = out
assert context.set_tensor_address("Y", out.value)
call("cudaDeviceSynchronize")
print("execute raw", a.batch, a.api, condition, flush=True)
assert context.execute_async_v3(handle)
call("cudaStreamSynchronize", stream)
call("cudaMemcpy", ctypes.c_void_p(actual.ctypes.data), out, actual.nbytes, 2)
print(
"actual",
actual.min(),
actual.max(),
"expected",
expected.min(),
expected.max(),
"max_error",
np.max(abs(actual - expected)),
flush=True,
)
for ptr in allocs.values():
call("cudaFree", ptr)
np.testing.assert_allclose(actual, expected, rtol=1e-5, atol=1e-4)
print("PASS raw", flush=True)
if not a.default_stream:
call("cudaStreamDestroy", stream)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("action", choices=("build", "run"))
parser.add_argument("--batch", type=int, default=4)
parser.add_argument("--height", type=int, default=19)
parser.add_argument("--width", type=int, default=31)
parser.add_argument("--opt-height", type=int, default=1024)
parser.add_argument("--opt-width", type=int, default=1024)
parser.add_argument("--api", action="store_true", help="Bypass the ONNX parser when building TensorRT")
parser.add_argument("--no-transpose", action="store_true")
parser.add_argument("--before", action="store_true", help="Move Transpose before If (control case)")
parser.add_argument("--default-stream", action="store_true")
parser.add_argument("--cudart", help="Path to libcudart.so when unavailable on the library search path")
parser.add_argument("--interpolation", choices=("cubic", "linear"), default="cubic")
parser.add_argument("--directory", type=Path, default=Path("dist/source-prepost/conditional-investigation/repro"))
args = parser.parse_args()
args.directory.mkdir(parents=True, exist_ok=True)
order = "plain" if args.no_transpose else ("transpose-before" if args.before else "transpose")
stem = f"b{args.batch}-{args.interpolation}-{order}"
args.onnx = args.directory / f"{stem}.onnx"
args.plan = args.directory / f"{stem}-{'api' if args.api else 'onnx'}.plan"
if args.action == "build":
onnx.save(model(args.batch, args.interpolation, not args.no_transpose, args.before), args.onnx)
build(args)
else:
run(args)
if __name__ == "__main__":
main()
Steps to reproduce
Install the dependencies in a Python 3.13 environment on a CUDA-capable Linux host:
python -m pip install tensorrt-cu13==11.2.1.2 numpy==2.5.1 \
onnx==1.22.0 onnxruntime==1.27.0 nvidia-cuda-runtime==13.3.29
For the 11.1 comparison, use tensorrt-cu13==11.1.0.106 in a separate environment. Locate the installed CUDA runtime and build/run as separate processes:
TRT_REPRO_CUDART=$(python -c 'import pathlib,sysconfig; print(next(pathlib.Path(sysconfig.get_paths()["purelib"]).rglob("libcudart.so.13")))')
python reproduce_trt_conditional_transpose.py build --api --directory repro
python reproduce_trt_conditional_transpose.py run --api --directory repro \
--cudart "$TRT_REPRO_CUDART"
An existing system libcudart.so.13 path can be supplied instead. The script also supports automatic library lookup when --cudart is omitted.
The default case uses these settings:
| Input / setting | Value |
|---|---|
| X | FP32 [4,1,H,W] |
| COND | Scalar BOOL |
| SIZE | INT32 [2] |
| X spatial profile min / opt / max | 1×1 / 1024×1024 / 1536×3072 |
| SIZE profile min / opt / max | 1×1 / 512×512 / 3072×3072 |
| Runtime X / SIZE | [4,1,19,31] / [19,33] |
| Synthetic X | (arange(4*19*31).reshape(4,1,19,31) % 113) * 0.1, FP32 |
| Resize | Cubic, align corners, coefficient −0.75 |
| TF32 / workspace limit | Disabled / 512 MiB |
Omit --api from both commands to reproduce through the ONNX parser. Use a separate directory when changing profile dimensions, since the filename does not encode the profile.
Expected behavior
TensorRT should match the ONNX Runtime CPU result within rtol=1e-5, atol=1e-4 for this valid input shape. The reference output for the first True condition has range 0.5502561…14.6497345.
Actual behavior
Engine construction succeeds, but the first inference produces range
−0.38888925…3.607245, with maximum absolute error 14.6497345:
execute raw 4 True True
actual -0.38888925 3.607245 expected 0.5502561 14.6497345 max_error 14.6497345
AssertionError: Not equal to tolerance rtol=1e-05, atol=0.0001
The B4 direct-API and ONNX paths produce these same values on both tested GPUs. A second fresh-process execution of the B4 direct-API engine on A10G reproduces the same failure.
Have you tried the latest release?: Yes
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes
- Vorherrschende Sprache
- C++
- Sterne
- 13.5k
- Forks
- 2.4k
- Ø Merge
- 6 T. 21 Std.
- Gemergte PRs (30 T.)
- 3
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Keine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus NVIDIA/TensorRT
-
Module:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 77/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyEvtl. vergeben @0z5a hat das vor 16 Tagen übernommen. OffenModule:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Polygraphy INT32 comparison can overflow and falsely pass at zero toleranceEvtl. vergeben @0z5a hat das vor 16 Tagen übernommen. OffenModule:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 6 Tagen
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodeEvtl. vergeben @gabortar hat das vor 39 Tagen übernommen. OffenModule:GraphSurgeon
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Polygraphy parse_meta accepts a shape with a missing bracketEvtl. vergeben @SashaMIT hat das vor 51 Tagen übernommen. OffenModule:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 86/100
Maintainer antworten meist innerhalb von 6 Tagen
Alle Issues in NVIDIA/TensorRT
Ähnliche Issues
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 91/100
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 69/100
maplibre/maplibre-native#5072 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 79/100
tenstorrent/tt-umd#4017 ·
Maintainer antworten meist innerhalb von 3 Tagen
-
difficulty:easy func:gui prio:low type:bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
NatronGitHub/Natron#1538 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
hyprwm/hyprsunset#103 · 1 Kommentar ·