Wrong results of TensorRT 10.14 / 11.4 static-shape engines when a batch-half swap (Slice+Slice+Concat) is fused with two reductions (FP32 too; dynamic shapes correct)

Ouverte
#4,893 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 6 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
32/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python, pytorch

Piste de recherche

Start with the supplied repro.py and its six-node static ONNX model, then run it with TensorRT and compare the static and dynamic engine outputs against the included NumPy reference. The issue reports the static fused Slice+Slice+Concat and reductions reading the wrong batch half, while the dynamic engine is correct. Done means identifying and fixing the static-shape result so both batch elements match the reference.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Module:Accuracy

Description

For the pattern

x_swapped = Concat(x[h:2h], x[0:h], axis=0)    # swap the two halves of the batch, h = N/2
y         = Reduce(x) + Reduce(x_swapped)      # e.g. ReduceMean, or LayerNorm(x) + LayerNorm(x_swapped)

Myelin fuses the Slice + Slice + Concat and both reductions into one kernel (__myl_MoveSlicSlicConcMeanMeanAdd_myl0_0 for the 6-node graph below). With a static input shape, the engine reads the second half of x_swapped from x[h:2h] instead of x[0:h]. It computes Reduce(x) + Reduce([x1, x1]) instead of Reduce(x) + Reduce([x1, x0]). The first half of the batch is correct and the second half is wrong.

  • This is not a precision problem. FP32 engines are wrong in the same way: TensorRT 10.14 with no precision flags, and TensorRT 11.4 strongly typed FP32. The wrong batch element matches Reduce([x1, x1]) to ~2e-8. FP16 engines, built with the FP16 flag or strongly typed, are also wrong.
  • An engine built from the same ONNX with a dynamic-shape profile (N = 2..32) is correct at N = 2 and N = 4. It contains a kernel with the same fused name; only the static-shape variant is wrong.
  • Any static N fails. On a static N = 4 engine, elements 2 and 3 are wrong.
  • Builder optimization levels 0-5 all give the same wrong result.
  • TensorRT 10.14.1.48, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 give bit-identical wrong numbers.
  • ONNX Runtime CPU (1.23.2, ORT_DISABLE_ALL) and a numpy float64 reference agree with each other to ~1e-7.

We found this in a transformer that implements cross-attention between two inputs by swapping the batch halves (torch.chunk(x, 2, 0) + torch.cat). A static batch-size engine was far less accurate than a dynamic-shape engine built from the same ONNX. The script below reduces the problem to the pattern itself.

Workaround: build with a dynamic-shape optimization profile, even if only one batch size is used at runtime.

Environment

TensorRT Version: 10.14.1.48.post1 and 11.4.0.106 (tensorrt_cu13 wheels, separate venvs); also reproduced with TensorRT-RTX 1.7.1.116 (tensorrt_rtx_cu13)

NVIDIA GPU: GeForce RTX 5090 Laptop GPU (sm_120, 24 GB)

NVIDIA Driver Version: 610.74 (CUDA UMD 13.3)

CUDA Version: 13.1.80 (nvidia-cuda-runtime wheel)

CUDNN Version: not used

Operating System: Windows 11 (10.0.26200, x64)

Python Version (if applicable): 3.12.11

PyTorch Version (if applicable): 2.7.1+cu128 (only to allocate device buffers and a stream)

Baremetal or Container (if so, version): Baremetal

Other: onnx 1.18.0, numpy 2.1.2

Relevant Files

Model link: none needed. The script builds both ONNX models in memory with onnx.helper:

  • A: 6 nodes with no weights and a static input x[2,16,8].
  • B: y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h])) on a dynamic-batch input x[N,1024,64]. The halves are computed from Shape(x), as torch.chunk exports them. B is built once with a static profile (N = 2) and once with a dynamic profile (2 / 8 / 32), and both are run at N = 2.

Steps To Reproduce

Commands or scripts:

pip install tensorrt numpy onnx torch      # or tensorrt_rtx
python repro.py          # TensorRT 10.x / 11.x (11.x: strongly typed FP32; 10.x: no precision flags)
python repro.py --rtx    # TensorRT-RTX

repro.py:

"""TensorRT static-shape engines compute y = Reduce(x) + Reduce(Concat(x[h:2h], x[0:h], axis=0)) wrongly: the second half
of the swapped tensor is read from x[h:2h] instead of x[0:h]. Dynamic-shape engines from the same ONNX are correct.
Requirements: tensorrt (10.x / 11.x) or tensorrt_rtx (--rtx), numpy, onnx, torch with CUDA (device buffers only)."""
import sys

import numpy as np
import onnx
import torch
from onnx import TensorProto as TP, helper as H, numpy_helper as NH

RTX = "--rtx" in sys.argv
if RTX:
    import tensorrt_rtx as trt
else:
    import tensorrt as trt
LOG = trt.Logger(trt.Logger.ERROR)
STRONG = RTX or int(trt.__version__.split(".")[0]) >= 11   # TRT 11 / RTX: strongly typed only


def i64(name, v):
    return NH.from_array(np.asarray(v, np.int64), name)


def model_a():
    """6 nodes, no weights, static x[2,16,8]."""
    nodes = [H.make_node("Slice", ["x", "zero", "one", "zero"], ["x_first"]),
             H.make_node("Slice", ["x", "one", "two", "zero"], ["x_second"]),
             H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
             H.make_node("ReduceMean", ["x"], ["m_x"], axes=[-1], keepdims=1),
             H.make_node("ReduceMean", ["x_swapped"], ["m_swapped"], axes=[-1], keepdims=1),
             H.make_node("Add", ["m_x", "m_swapped"], ["y"])]
    g = H.make_graph(nodes, "a", [H.make_tensor_value_info("x", TP.FLOAT, [2, 16, 8])],
                     [H.make_tensor_value_info("y", TP.FLOAT, [2, 16, 1])],
                     [i64("zero", [0]), i64("one", [1]), i64("two", [2])])
    m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
    m.ir_version = 8
    return m


def model_b(g1, b1, g2, b2):
    """y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h])), x[N,1024,64], halves from Shape like torch.chunk."""
    nodes = [H.make_node("Shape", ["x"], ["shape"]), H.make_node("Gather", ["shape", "zero"], ["n"], axis=0),
             H.make_node("Div", ["n", "two"], ["h"]), H.make_node("Mul", ["h", "two"], ["n2"]),
             H.make_node("Slice", ["x", "zero", "h", "zero"], ["x_first"]),
             H.make_node("Slice", ["x", "h", "n2", "zero"], ["x_second"]),
             H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
             H.make_node("LayerNormalization", ["x", "g1", "b1"], ["ln1"], axis=-1, epsilon=1e-6),
             H.make_node("LayerNormalization", ["x_swapped", "g2", "b2"], ["ln2"], axis=-1, epsilon=1e-6),
             H.make_node("Add", ["ln1", "ln2"], ["y"])]
    f32 = np.float32
    g = H.make_graph(nodes, "b", [H.make_tensor_value_info("x", TP.FLOAT, ["N", 1024, 64])],
                     [H.make_tensor_value_info("y", TP.FLOAT, ["N", 1024, 64])],
                     [i64("zero", [0]), i64("two", [2]), NH.from_array(g1.astype(f32), "g1"),
                      NH.from_array(b1.astype(f32), "b1"), NH.from_array(g2.astype(f32), "g2"),
                      NH.from_array(b2.astype(f32), "b2")])
    m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
    m.ir_version = 8
    return m


def run(model, x, profile=None):
    b = trt.Builder(LOG)
    flags = 0 if RTX else (1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED) if STRONG else 0)
    net = b.create_network(flags)
    p = trt.OnnxParser(net, LOG)
    assert p.parse(model.SerializeToString()), [str(p.get_error(i)) for i in range(p.num_errors)]
    cfg = b.create_builder_config()
    if profile:
        rest = list(net.get_input(0).shape)[1:]
        pr = b.create_optimization_profile()
        pr.set_shape("x", *[[n] + rest for n in profile])
        cfg.add_optimization_profile(pr)
    eng = trt.Runtime(LOG).deserialize_cuda_engine(b.build_serialized_network(net, cfg))
    ctx = eng.create_execution_context(eng.create_runtime_config()) if RTX else eng.create_execution_context()
    xin = torch.from_numpy(x).cuda()
    ctx.set_input_shape("x", tuple(xin.shape))
    y = torch.empty(tuple(ctx.get_tensor_shape("y")), dtype=torch.float32, device="cuda")
    ctx.set_tensor_address("x", xin.data_ptr())
    ctx.set_tensor_address("y", y.data_ptr())
    s = torch.cuda.Stream()
    torch.cuda.synchronize()
    assert ctx.execute_async_v3(s.cuda_stream)
    s.synchronize()
    return y.cpu().numpy().astype(np.float64)


def err(y, ref):  # mean abs error per batch element
    return np.abs(y - ref).reshape(len(y), -1).mean(1)


def ln(v, g, b):
    mu = v.mean(-1, keepdims=True)
    return (v - mu) / np.sqrt(((v - mu) ** 2).mean(-1, keepdims=True) + 1e-6) * g + b


print(("TensorRT-RTX " if RTX else "TensorRT ") + trt.__version__, "|", torch.cuda.get_device_name(0),
      "| strongly typed FP32" if STRONG else "| FP32, no precision flags")
x = np.random.default_rng(0).standard_normal((2, 16, 8)).astype(np.float32)
xd = x.astype(np.float64)
ref = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[:1]]).mean(-1, keepdims=True)
bug = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[1:]]).mean(-1, keepdims=True)
y = run(model_a(), x)
print("A  static N=2        : mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))

rng = np.random.default_rng(7)
g1, b1, g2, b2 = 0.5 + rng.random(64), 0.2 * rng.standard_normal(64), 0.5 + rng.random(64), 0.2 * rng.standard_normal(64)
mb = model_b(g1, b1, g2, b2)
x = np.random.default_rng(1).standard_normal((2, 1024, 64)).astype(np.float32)
xd = x.astype(np.float64)
ref = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[:1]]), g2, b2)
bug = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[1:]]), g2, b2)
for name, prof in (("static N=2  ", (2, 2, 2)), ("dynamic 2..32", (2, 8, 32))):
    y = run(mb, x, prof)
    print(f"B  {name}, run N=2: mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))

Output. TensorRT 10.14.1.48.post1, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 print the same numbers; only the first line differs.

TensorRT 10.14.1.48.post1 | NVIDIA GeForce RTX 5090 Laptop GPU | FP32, no precision flags
A  static N=2        : mean|y-ref| per batch element [1.88265403e-08 4.11452092e-01]  | vs 'swapped=[x1,x1]' [1.88265403e-08 1.91575964e-08]
B  static N=2  , run N=2: mean|y-ref| per batch element [7.75532025e-08 1.15770741e+00]  | vs 'swapped=[x1,x1]' [7.75532025e-08 9.34008044e-08]
B  dynamic 2..32, run N=2: mean|y-ref| per batch element [7.75532025e-08 7.76504523e-08]  | vs 'swapped=[x1,x1]' [7.75532025e-08 1.15770741e+00]
TensorRT 11.4.0.106 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
TensorRT-RTX 1.7.1.116 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
(remaining lines identical to the above)

How to read the output:

  • Static engines (A, and B static): batch element 1 is off by 0.41 / 1.16 from the reference, but matches the "swapped = [x1, x1]" result to ~1e-7.
  • Dynamic engine (B dynamic): correct.

Have you tried the latest release?: Yes. 11.4.0.106 is the latest tensorrt-cu13 wheel, and 1.7.1.116 is the latest tensorrt-rtx wheel.

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes. ONNX Runtime CPU gives the correct result for both models (mean abs error vs the numpy reference: A 1.9e-8, B 9.9e-8). The TensorRT dynamic-shape engine is also correct.

Langage dominant
C++
Étoiles
13.5k
Forks
2.4k
Merge moyen
6 j 21 h
PR mergées (30 j)
3

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/TensorRT

Toutes les issues de NVIDIA/TensorRT

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.