Wrong results of TensorRT 10.14 / 11.4 static-shape engines when a batch-half swap (Slice+Slice+Concat) is fused with two reductions (FP32 too; dynamic shapes correct)
Les mainteneurs répondent en général sous 6 jours
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Accessibilité débutants
- 32/100
- Type d'issue
- Bug
- Clarté
- Clairement spécifiée
- Activité
- Active
- Stack technique
- python, pytorch
- Domaine
- backend, machine-learning
Piste de recherche
Start with the supplied repro.py and its six-node static ONNX model, then run it with TensorRT and compare the static and dynamic engine outputs against the included NumPy reference. The issue reports the static fused Slice+Slice+Concat and reductions reading the wrong batch half, while the dynamic engine is correct. Done means identifying and fixing the static-shape result so both batch elements match the reference.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Description
For the pattern
x_swapped = Concat(x[h:2h], x[0:h], axis=0) # swap the two halves of the batch, h = N/2
y = Reduce(x) + Reduce(x_swapped) # e.g. ReduceMean, or LayerNorm(x) + LayerNorm(x_swapped)
Myelin fuses the Slice + Slice + Concat and both reductions into one kernel (__myl_MoveSlicSlicConcMeanMeanAdd_myl0_0 for the 6-node graph below). With a static input shape, the engine reads the second half of x_swapped from x[h:2h] instead of x[0:h]. It computes Reduce(x) + Reduce([x1, x1]) instead of Reduce(x) + Reduce([x1, x0]). The first half of the batch is correct and the second half is wrong.
- This is not a precision problem. FP32 engines are wrong in the same way: TensorRT 10.14 with no precision flags, and TensorRT 11.4 strongly typed FP32. The wrong batch element matches
Reduce([x1, x1])to ~2e-8. FP16 engines, built with the FP16 flag or strongly typed, are also wrong. - An engine built from the same ONNX with a dynamic-shape profile (N = 2..32) is correct at N = 2 and N = 4. It contains a kernel with the same fused name; only the static-shape variant is wrong.
- Any static N fails. On a static N = 4 engine, elements 2 and 3 are wrong.
- Builder optimization levels 0-5 all give the same wrong result.
- TensorRT 10.14.1.48, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 give bit-identical wrong numbers.
- ONNX Runtime CPU (1.23.2,
ORT_DISABLE_ALL) and a numpy float64 reference agree with each other to ~1e-7.
We found this in a transformer that implements cross-attention between two inputs by swapping the batch halves (torch.chunk(x, 2, 0) + torch.cat). A static batch-size engine was far less accurate than a dynamic-shape engine built from the same ONNX. The script below reduces the problem to the pattern itself.
Workaround: build with a dynamic-shape optimization profile, even if only one batch size is used at runtime.
Environment
TensorRT Version: 10.14.1.48.post1 and 11.4.0.106 (tensorrt_cu13 wheels, separate venvs); also reproduced with TensorRT-RTX 1.7.1.116 (tensorrt_rtx_cu13)
NVIDIA GPU: GeForce RTX 5090 Laptop GPU (sm_120, 24 GB)
NVIDIA Driver Version: 610.74 (CUDA UMD 13.3)
CUDA Version: 13.1.80 (nvidia-cuda-runtime wheel)
CUDNN Version: not used
Operating System: Windows 11 (10.0.26200, x64)
Python Version (if applicable): 3.12.11
PyTorch Version (if applicable): 2.7.1+cu128 (only to allocate device buffers and a stream)
Baremetal or Container (if so, version): Baremetal
Other: onnx 1.18.0, numpy 2.1.2
Relevant Files
Model link: none needed. The script builds both ONNX models in memory with onnx.helper:
- A: 6 nodes with no weights and a static input
x[2,16,8]. - B:
y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h]))on a dynamic-batch inputx[N,1024,64]. The halves are computed fromShape(x), astorch.chunkexports them. B is built once with a static profile (N = 2) and once with a dynamic profile (2 / 8 / 32), and both are run at N = 2.
Steps To Reproduce
Commands or scripts:
pip install tensorrt numpy onnx torch # or tensorrt_rtx
python repro.py # TensorRT 10.x / 11.x (11.x: strongly typed FP32; 10.x: no precision flags)
python repro.py --rtx # TensorRT-RTX
repro.py:
"""TensorRT static-shape engines compute y = Reduce(x) + Reduce(Concat(x[h:2h], x[0:h], axis=0)) wrongly: the second half
of the swapped tensor is read from x[h:2h] instead of x[0:h]. Dynamic-shape engines from the same ONNX are correct.
Requirements: tensorrt (10.x / 11.x) or tensorrt_rtx (--rtx), numpy, onnx, torch with CUDA (device buffers only)."""
import sys
import numpy as np
import onnx
import torch
from onnx import TensorProto as TP, helper as H, numpy_helper as NH
RTX = "--rtx" in sys.argv
if RTX:
import tensorrt_rtx as trt
else:
import tensorrt as trt
LOG = trt.Logger(trt.Logger.ERROR)
STRONG = RTX or int(trt.__version__.split(".")[0]) >= 11 # TRT 11 / RTX: strongly typed only
def i64(name, v):
return NH.from_array(np.asarray(v, np.int64), name)
def model_a():
"""6 nodes, no weights, static x[2,16,8]."""
nodes = [H.make_node("Slice", ["x", "zero", "one", "zero"], ["x_first"]),
H.make_node("Slice", ["x", "one", "two", "zero"], ["x_second"]),
H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
H.make_node("ReduceMean", ["x"], ["m_x"], axes=[-1], keepdims=1),
H.make_node("ReduceMean", ["x_swapped"], ["m_swapped"], axes=[-1], keepdims=1),
H.make_node("Add", ["m_x", "m_swapped"], ["y"])]
g = H.make_graph(nodes, "a", [H.make_tensor_value_info("x", TP.FLOAT, [2, 16, 8])],
[H.make_tensor_value_info("y", TP.FLOAT, [2, 16, 1])],
[i64("zero", [0]), i64("one", [1]), i64("two", [2])])
m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
m.ir_version = 8
return m
def model_b(g1, b1, g2, b2):
"""y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h])), x[N,1024,64], halves from Shape like torch.chunk."""
nodes = [H.make_node("Shape", ["x"], ["shape"]), H.make_node("Gather", ["shape", "zero"], ["n"], axis=0),
H.make_node("Div", ["n", "two"], ["h"]), H.make_node("Mul", ["h", "two"], ["n2"]),
H.make_node("Slice", ["x", "zero", "h", "zero"], ["x_first"]),
H.make_node("Slice", ["x", "h", "n2", "zero"], ["x_second"]),
H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
H.make_node("LayerNormalization", ["x", "g1", "b1"], ["ln1"], axis=-1, epsilon=1e-6),
H.make_node("LayerNormalization", ["x_swapped", "g2", "b2"], ["ln2"], axis=-1, epsilon=1e-6),
H.make_node("Add", ["ln1", "ln2"], ["y"])]
f32 = np.float32
g = H.make_graph(nodes, "b", [H.make_tensor_value_info("x", TP.FLOAT, ["N", 1024, 64])],
[H.make_tensor_value_info("y", TP.FLOAT, ["N", 1024, 64])],
[i64("zero", [0]), i64("two", [2]), NH.from_array(g1.astype(f32), "g1"),
NH.from_array(b1.astype(f32), "b1"), NH.from_array(g2.astype(f32), "g2"),
NH.from_array(b2.astype(f32), "b2")])
m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
m.ir_version = 8
return m
def run(model, x, profile=None):
b = trt.Builder(LOG)
flags = 0 if RTX else (1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED) if STRONG else 0)
net = b.create_network(flags)
p = trt.OnnxParser(net, LOG)
assert p.parse(model.SerializeToString()), [str(p.get_error(i)) for i in range(p.num_errors)]
cfg = b.create_builder_config()
if profile:
rest = list(net.get_input(0).shape)[1:]
pr = b.create_optimization_profile()
pr.set_shape("x", *[[n] + rest for n in profile])
cfg.add_optimization_profile(pr)
eng = trt.Runtime(LOG).deserialize_cuda_engine(b.build_serialized_network(net, cfg))
ctx = eng.create_execution_context(eng.create_runtime_config()) if RTX else eng.create_execution_context()
xin = torch.from_numpy(x).cuda()
ctx.set_input_shape("x", tuple(xin.shape))
y = torch.empty(tuple(ctx.get_tensor_shape("y")), dtype=torch.float32, device="cuda")
ctx.set_tensor_address("x", xin.data_ptr())
ctx.set_tensor_address("y", y.data_ptr())
s = torch.cuda.Stream()
torch.cuda.synchronize()
assert ctx.execute_async_v3(s.cuda_stream)
s.synchronize()
return y.cpu().numpy().astype(np.float64)
def err(y, ref): # mean abs error per batch element
return np.abs(y - ref).reshape(len(y), -1).mean(1)
def ln(v, g, b):
mu = v.mean(-1, keepdims=True)
return (v - mu) / np.sqrt(((v - mu) ** 2).mean(-1, keepdims=True) + 1e-6) * g + b
print(("TensorRT-RTX " if RTX else "TensorRT ") + trt.__version__, "|", torch.cuda.get_device_name(0),
"| strongly typed FP32" if STRONG else "| FP32, no precision flags")
x = np.random.default_rng(0).standard_normal((2, 16, 8)).astype(np.float32)
xd = x.astype(np.float64)
ref = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[:1]]).mean(-1, keepdims=True)
bug = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[1:]]).mean(-1, keepdims=True)
y = run(model_a(), x)
print("A static N=2 : mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))
rng = np.random.default_rng(7)
g1, b1, g2, b2 = 0.5 + rng.random(64), 0.2 * rng.standard_normal(64), 0.5 + rng.random(64), 0.2 * rng.standard_normal(64)
mb = model_b(g1, b1, g2, b2)
x = np.random.default_rng(1).standard_normal((2, 1024, 64)).astype(np.float32)
xd = x.astype(np.float64)
ref = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[:1]]), g2, b2)
bug = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[1:]]), g2, b2)
for name, prof in (("static N=2 ", (2, 2, 2)), ("dynamic 2..32", (2, 8, 32))):
y = run(mb, x, prof)
print(f"B {name}, run N=2: mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))
Output. TensorRT 10.14.1.48.post1, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 print the same numbers; only the first line differs.
TensorRT 10.14.1.48.post1 | NVIDIA GeForce RTX 5090 Laptop GPU | FP32, no precision flags
A static N=2 : mean|y-ref| per batch element [1.88265403e-08 4.11452092e-01] | vs 'swapped=[x1,x1]' [1.88265403e-08 1.91575964e-08]
B static N=2 , run N=2: mean|y-ref| per batch element [7.75532025e-08 1.15770741e+00] | vs 'swapped=[x1,x1]' [7.75532025e-08 9.34008044e-08]
B dynamic 2..32, run N=2: mean|y-ref| per batch element [7.75532025e-08 7.76504523e-08] | vs 'swapped=[x1,x1]' [7.75532025e-08 1.15770741e+00]
TensorRT 11.4.0.106 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
TensorRT-RTX 1.7.1.116 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
(remaining lines identical to the above)
How to read the output:
- Static engines (A, and B static): batch element 1 is off by 0.41 / 1.16 from the reference, but matches the "swapped = [x1, x1]" result to ~1e-7.
- Dynamic engine (B dynamic): correct.
Have you tried the latest release?: Yes. 11.4.0.106 is the latest tensorrt-cu13 wheel, and 1.7.1.116 is the latest tensorrt-rtx wheel.
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes. ONNX Runtime CPU gives the correct result for both models (mean abs error vs the numpy reference: A 1.9e-8, B 9.9e-8). The TensorRT dynamic-shape engine is also correct.
- Langage dominant
- C++
- Étoiles
- 13.5k
- Forks
- 2.4k
- Merge moyen
- 6 j 21 h
- PR mergées (30 j)
- 3
Préparer son environnement
- Aucun Dockerfile ni fichier Docker Compose
- Aucun modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de NVIDIA/TensorRT
-
Module:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 77/100
Les mainteneurs répondent en général sous 6 jours
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyPeut-être pris @0z5a l’a pris il y a 16 jours. OuverteModule:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 6 jours
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerancePeut-être pris @0z5a l’a pris il y a 16 jours. OuverteModule:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 6 jours
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodePeut-être pris @gabortar l’a pris il y a 39 jours. OuverteModule:GraphSurgeon
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
Les mainteneurs répondent en général sous 6 jours
-
Polygraphy parse_meta accepts a shape with a missing bracketPeut-être pris @SashaMIT l’a pris il y a 51 jours. OuverteModule:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 86/100
Les mainteneurs répondent en général sous 6 jours
Toutes les issues de NVIDIA/TensorRT
Issues similaires
-
Difficulté 1/5 Moins d'une heure Accessibilité débutants 91/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 69/100
maplibre/maplibre-native#5072 ·
Les mainteneurs répondent en général sous 1 jour
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 79/100
tenstorrent/tt-umd#4017 ·
Les mainteneurs répondent en général sous 3 jours
-
difficulty:easy func:gui prio:low type:bug
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
NatronGitHub/Natron#1538 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
hyprwm/hyprsunset#103 · 1 commentaire ·