Wrong results of TensorRT 10.14 / 11.4 static-shape engines when a batch-half swap (Slice+Slice+Concat) is fused with two reductions (FP32 too; dynamic shapes correct)
I maintainer di solito rispondono entro 6 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 32/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- python, pytorch
- Ambito
- backend, machine-learning
Direzione di ricerca
Start with the supplied repro.py and its six-node static ONNX model, then run it with TensorRT and compare the static and dynamic engine outputs against the included NumPy reference. The issue reports the static fused Slice+Slice+Concat and reductions reading the wrong batch half, while the dynamic engine is correct. Done means identifying and fixing the static-shape result so both batch elements match the reference.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Description
For the pattern
x_swapped = Concat(x[h:2h], x[0:h], axis=0) # swap the two halves of the batch, h = N/2
y = Reduce(x) + Reduce(x_swapped) # e.g. ReduceMean, or LayerNorm(x) + LayerNorm(x_swapped)
Myelin fuses the Slice + Slice + Concat and both reductions into one kernel (__myl_MoveSlicSlicConcMeanMeanAdd_myl0_0 for the 6-node graph below). With a static input shape, the engine reads the second half of x_swapped from x[h:2h] instead of x[0:h]. It computes Reduce(x) + Reduce([x1, x1]) instead of Reduce(x) + Reduce([x1, x0]). The first half of the batch is correct and the second half is wrong.
- This is not a precision problem. FP32 engines are wrong in the same way: TensorRT 10.14 with no precision flags, and TensorRT 11.4 strongly typed FP32. The wrong batch element matches
Reduce([x1, x1])to ~2e-8. FP16 engines, built with the FP16 flag or strongly typed, are also wrong. - An engine built from the same ONNX with a dynamic-shape profile (N = 2..32) is correct at N = 2 and N = 4. It contains a kernel with the same fused name; only the static-shape variant is wrong.
- Any static N fails. On a static N = 4 engine, elements 2 and 3 are wrong.
- Builder optimization levels 0-5 all give the same wrong result.
- TensorRT 10.14.1.48, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 give bit-identical wrong numbers.
- ONNX Runtime CPU (1.23.2,
ORT_DISABLE_ALL) and a numpy float64 reference agree with each other to ~1e-7.
We found this in a transformer that implements cross-attention between two inputs by swapping the batch halves (torch.chunk(x, 2, 0) + torch.cat). A static batch-size engine was far less accurate than a dynamic-shape engine built from the same ONNX. The script below reduces the problem to the pattern itself.
Workaround: build with a dynamic-shape optimization profile, even if only one batch size is used at runtime.
Environment
TensorRT Version: 10.14.1.48.post1 and 11.4.0.106 (tensorrt_cu13 wheels, separate venvs); also reproduced with TensorRT-RTX 1.7.1.116 (tensorrt_rtx_cu13)
NVIDIA GPU: GeForce RTX 5090 Laptop GPU (sm_120, 24 GB)
NVIDIA Driver Version: 610.74 (CUDA UMD 13.3)
CUDA Version: 13.1.80 (nvidia-cuda-runtime wheel)
CUDNN Version: not used
Operating System: Windows 11 (10.0.26200, x64)
Python Version (if applicable): 3.12.11
PyTorch Version (if applicable): 2.7.1+cu128 (only to allocate device buffers and a stream)
Baremetal or Container (if so, version): Baremetal
Other: onnx 1.18.0, numpy 2.1.2
Relevant Files
Model link: none needed. The script builds both ONNX models in memory with onnx.helper:
- A: 6 nodes with no weights and a static input
x[2,16,8]. - B:
y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h]))on a dynamic-batch inputx[N,1024,64]. The halves are computed fromShape(x), astorch.chunkexports them. B is built once with a static profile (N = 2) and once with a dynamic profile (2 / 8 / 32), and both are run at N = 2.
Steps To Reproduce
Commands or scripts:
pip install tensorrt numpy onnx torch # or tensorrt_rtx
python repro.py # TensorRT 10.x / 11.x (11.x: strongly typed FP32; 10.x: no precision flags)
python repro.py --rtx # TensorRT-RTX
repro.py:
"""TensorRT static-shape engines compute y = Reduce(x) + Reduce(Concat(x[h:2h], x[0:h], axis=0)) wrongly: the second half
of the swapped tensor is read from x[h:2h] instead of x[0:h]. Dynamic-shape engines from the same ONNX are correct.
Requirements: tensorrt (10.x / 11.x) or tensorrt_rtx (--rtx), numpy, onnx, torch with CUDA (device buffers only)."""
import sys
import numpy as np
import onnx
import torch
from onnx import TensorProto as TP, helper as H, numpy_helper as NH
RTX = "--rtx" in sys.argv
if RTX:
import tensorrt_rtx as trt
else:
import tensorrt as trt
LOG = trt.Logger(trt.Logger.ERROR)
STRONG = RTX or int(trt.__version__.split(".")[0]) >= 11 # TRT 11 / RTX: strongly typed only
def i64(name, v):
return NH.from_array(np.asarray(v, np.int64), name)
def model_a():
"""6 nodes, no weights, static x[2,16,8]."""
nodes = [H.make_node("Slice", ["x", "zero", "one", "zero"], ["x_first"]),
H.make_node("Slice", ["x", "one", "two", "zero"], ["x_second"]),
H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
H.make_node("ReduceMean", ["x"], ["m_x"], axes=[-1], keepdims=1),
H.make_node("ReduceMean", ["x_swapped"], ["m_swapped"], axes=[-1], keepdims=1),
H.make_node("Add", ["m_x", "m_swapped"], ["y"])]
g = H.make_graph(nodes, "a", [H.make_tensor_value_info("x", TP.FLOAT, [2, 16, 8])],
[H.make_tensor_value_info("y", TP.FLOAT, [2, 16, 1])],
[i64("zero", [0]), i64("one", [1]), i64("two", [2])])
m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
m.ir_version = 8
return m
def model_b(g1, b1, g2, b2):
"""y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h])), x[N,1024,64], halves from Shape like torch.chunk."""
nodes = [H.make_node("Shape", ["x"], ["shape"]), H.make_node("Gather", ["shape", "zero"], ["n"], axis=0),
H.make_node("Div", ["n", "two"], ["h"]), H.make_node("Mul", ["h", "two"], ["n2"]),
H.make_node("Slice", ["x", "zero", "h", "zero"], ["x_first"]),
H.make_node("Slice", ["x", "h", "n2", "zero"], ["x_second"]),
H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
H.make_node("LayerNormalization", ["x", "g1", "b1"], ["ln1"], axis=-1, epsilon=1e-6),
H.make_node("LayerNormalization", ["x_swapped", "g2", "b2"], ["ln2"], axis=-1, epsilon=1e-6),
H.make_node("Add", ["ln1", "ln2"], ["y"])]
f32 = np.float32
g = H.make_graph(nodes, "b", [H.make_tensor_value_info("x", TP.FLOAT, ["N", 1024, 64])],
[H.make_tensor_value_info("y", TP.FLOAT, ["N", 1024, 64])],
[i64("zero", [0]), i64("two", [2]), NH.from_array(g1.astype(f32), "g1"),
NH.from_array(b1.astype(f32), "b1"), NH.from_array(g2.astype(f32), "g2"),
NH.from_array(b2.astype(f32), "b2")])
m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
m.ir_version = 8
return m
def run(model, x, profile=None):
b = trt.Builder(LOG)
flags = 0 if RTX else (1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED) if STRONG else 0)
net = b.create_network(flags)
p = trt.OnnxParser(net, LOG)
assert p.parse(model.SerializeToString()), [str(p.get_error(i)) for i in range(p.num_errors)]
cfg = b.create_builder_config()
if profile:
rest = list(net.get_input(0).shape)[1:]
pr = b.create_optimization_profile()
pr.set_shape("x", *[[n] + rest for n in profile])
cfg.add_optimization_profile(pr)
eng = trt.Runtime(LOG).deserialize_cuda_engine(b.build_serialized_network(net, cfg))
ctx = eng.create_execution_context(eng.create_runtime_config()) if RTX else eng.create_execution_context()
xin = torch.from_numpy(x).cuda()
ctx.set_input_shape("x", tuple(xin.shape))
y = torch.empty(tuple(ctx.get_tensor_shape("y")), dtype=torch.float32, device="cuda")
ctx.set_tensor_address("x", xin.data_ptr())
ctx.set_tensor_address("y", y.data_ptr())
s = torch.cuda.Stream()
torch.cuda.synchronize()
assert ctx.execute_async_v3(s.cuda_stream)
s.synchronize()
return y.cpu().numpy().astype(np.float64)
def err(y, ref): # mean abs error per batch element
return np.abs(y - ref).reshape(len(y), -1).mean(1)
def ln(v, g, b):
mu = v.mean(-1, keepdims=True)
return (v - mu) / np.sqrt(((v - mu) ** 2).mean(-1, keepdims=True) + 1e-6) * g + b
print(("TensorRT-RTX " if RTX else "TensorRT ") + trt.__version__, "|", torch.cuda.get_device_name(0),
"| strongly typed FP32" if STRONG else "| FP32, no precision flags")
x = np.random.default_rng(0).standard_normal((2, 16, 8)).astype(np.float32)
xd = x.astype(np.float64)
ref = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[:1]]).mean(-1, keepdims=True)
bug = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[1:]]).mean(-1, keepdims=True)
y = run(model_a(), x)
print("A static N=2 : mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))
rng = np.random.default_rng(7)
g1, b1, g2, b2 = 0.5 + rng.random(64), 0.2 * rng.standard_normal(64), 0.5 + rng.random(64), 0.2 * rng.standard_normal(64)
mb = model_b(g1, b1, g2, b2)
x = np.random.default_rng(1).standard_normal((2, 1024, 64)).astype(np.float32)
xd = x.astype(np.float64)
ref = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[:1]]), g2, b2)
bug = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[1:]]), g2, b2)
for name, prof in (("static N=2 ", (2, 2, 2)), ("dynamic 2..32", (2, 8, 32))):
y = run(mb, x, prof)
print(f"B {name}, run N=2: mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))
Output. TensorRT 10.14.1.48.post1, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 print the same numbers; only the first line differs.
TensorRT 10.14.1.48.post1 | NVIDIA GeForce RTX 5090 Laptop GPU | FP32, no precision flags
A static N=2 : mean|y-ref| per batch element [1.88265403e-08 4.11452092e-01] | vs 'swapped=[x1,x1]' [1.88265403e-08 1.91575964e-08]
B static N=2 , run N=2: mean|y-ref| per batch element [7.75532025e-08 1.15770741e+00] | vs 'swapped=[x1,x1]' [7.75532025e-08 9.34008044e-08]
B dynamic 2..32, run N=2: mean|y-ref| per batch element [7.75532025e-08 7.76504523e-08] | vs 'swapped=[x1,x1]' [7.75532025e-08 1.15770741e+00]
TensorRT 11.4.0.106 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
TensorRT-RTX 1.7.1.116 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
(remaining lines identical to the above)
How to read the output:
- Static engines (A, and B static): batch element 1 is off by 0.41 / 1.16 from the reference, but matches the "swapped = [x1, x1]" result to ~1e-7.
- Dynamic engine (B dynamic): correct.
Have you tried the latest release?: Yes. 11.4.0.106 is the latest tensorrt-cu13 wheel, and 1.7.1.116 is the latest tensorrt-rtx wheel.
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes. ONNX Runtime CPU gives the correct result for both models (mean abs error vs the numpy reference: A 1.9e-8, B 9.9e-8). The TensorRT dynamic-shape engine is also correct.
- Lingua principale
- C++
- Stelle
- 13.5k
- Fork
- 2.4k
- Merge medio
- 6g 21h
- PR unite (30g)
- 3
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di NVIDIA/TensorRT
-
Module:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyForse già presa @0z5a l’ha presa 16 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy INT32 comparison can overflow and falsely pass at zero toleranceForse già presa @0z5a l’ha presa 16 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 6 giorni
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodeForse già presa @gabortar l’ha presa 39 giorni fa. ApertaModule:GraphSurgeon
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy parse_meta accepts a shape with a missing bracketForse già presa @SashaMIT l’ha presa 51 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
I maintainer di solito rispondono entro 6 giorni
Tutte le issue di NVIDIA/TensorRT
Issue simili
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 91/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
maplibre/maplibre-native#5072 ·
I maintainer di solito rispondono entro 1 giorno
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 79/100
tenstorrent/tt-umd#4017 ·
I maintainer di solito rispondono entro 3 giorni
-
difficulty:easy func:gui prio:low type:bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
NatronGitHub/Natron#1538 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
hyprwm/hyprsunset#103 · 1 commento ·