Wrong results of TensorRT 10.14 / 11.4 static-shape engines when a batch-half swap (Slice+Slice+Concat) is fused with two reductions (FP32 too; dynamic shapes correct)

未关闭
#4,893 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 6 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
32/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python, pytorch

调研方向

Start with the supplied repro.py and its six-node static ONNX model, then run it with TensorRT and compare the static and dynamic engine outputs against the included NumPy reference. The issue reports the static fused Slice+Slice+Concat and reductions reading the wrong batch half, while the dynamic engine is correct. Done means identifying and fixing the static-shape result so both batch elements match the reference.

由索引模型根据 Issue 内容生成。

描述

Module:Accuracy

Description

For the pattern

x_swapped = Concat(x[h:2h], x[0:h], axis=0)    # swap the two halves of the batch, h = N/2
y         = Reduce(x) + Reduce(x_swapped)      # e.g. ReduceMean, or LayerNorm(x) + LayerNorm(x_swapped)

Myelin fuses the Slice + Slice + Concat and both reductions into one kernel (__myl_MoveSlicSlicConcMeanMeanAdd_myl0_0 for the 6-node graph below). With a static input shape, the engine reads the second half of x_swapped from x[h:2h] instead of x[0:h]. It computes Reduce(x) + Reduce([x1, x1]) instead of Reduce(x) + Reduce([x1, x0]). The first half of the batch is correct and the second half is wrong.

  • This is not a precision problem. FP32 engines are wrong in the same way: TensorRT 10.14 with no precision flags, and TensorRT 11.4 strongly typed FP32. The wrong batch element matches Reduce([x1, x1]) to ~2e-8. FP16 engines, built with the FP16 flag or strongly typed, are also wrong.
  • An engine built from the same ONNX with a dynamic-shape profile (N = 2..32) is correct at N = 2 and N = 4. It contains a kernel with the same fused name; only the static-shape variant is wrong.
  • Any static N fails. On a static N = 4 engine, elements 2 and 3 are wrong.
  • Builder optimization levels 0-5 all give the same wrong result.
  • TensorRT 10.14.1.48, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 give bit-identical wrong numbers.
  • ONNX Runtime CPU (1.23.2, ORT_DISABLE_ALL) and a numpy float64 reference agree with each other to ~1e-7.

We found this in a transformer that implements cross-attention between two inputs by swapping the batch halves (torch.chunk(x, 2, 0) + torch.cat). A static batch-size engine was far less accurate than a dynamic-shape engine built from the same ONNX. The script below reduces the problem to the pattern itself.

Workaround: build with a dynamic-shape optimization profile, even if only one batch size is used at runtime.

Environment

TensorRT Version: 10.14.1.48.post1 and 11.4.0.106 (tensorrt_cu13 wheels, separate venvs); also reproduced with TensorRT-RTX 1.7.1.116 (tensorrt_rtx_cu13)

NVIDIA GPU: GeForce RTX 5090 Laptop GPU (sm_120, 24 GB)

NVIDIA Driver Version: 610.74 (CUDA UMD 13.3)

CUDA Version: 13.1.80 (nvidia-cuda-runtime wheel)

CUDNN Version: not used

Operating System: Windows 11 (10.0.26200, x64)

Python Version (if applicable): 3.12.11

PyTorch Version (if applicable): 2.7.1+cu128 (only to allocate device buffers and a stream)

Baremetal or Container (if so, version): Baremetal

Other: onnx 1.18.0, numpy 2.1.2

Relevant Files

Model link: none needed. The script builds both ONNX models in memory with onnx.helper:

  • A: 6 nodes with no weights and a static input x[2,16,8].
  • B: y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h])) on a dynamic-batch input x[N,1024,64]. The halves are computed from Shape(x), as torch.chunk exports them. B is built once with a static profile (N = 2) and once with a dynamic profile (2 / 8 / 32), and both are run at N = 2.

Steps To Reproduce

Commands or scripts:

pip install tensorrt numpy onnx torch      # or tensorrt_rtx
python repro.py          # TensorRT 10.x / 11.x (11.x: strongly typed FP32; 10.x: no precision flags)
python repro.py --rtx    # TensorRT-RTX

repro.py:

"""TensorRT static-shape engines compute y = Reduce(x) + Reduce(Concat(x[h:2h], x[0:h], axis=0)) wrongly: the second half
of the swapped tensor is read from x[h:2h] instead of x[0:h]. Dynamic-shape engines from the same ONNX are correct.
Requirements: tensorrt (10.x / 11.x) or tensorrt_rtx (--rtx), numpy, onnx, torch with CUDA (device buffers only)."""
import sys

import numpy as np
import onnx
import torch
from onnx import TensorProto as TP, helper as H, numpy_helper as NH

RTX = "--rtx" in sys.argv
if RTX:
    import tensorrt_rtx as trt
else:
    import tensorrt as trt
LOG = trt.Logger(trt.Logger.ERROR)
STRONG = RTX or int(trt.__version__.split(".")[0]) >= 11   # TRT 11 / RTX: strongly typed only


def i64(name, v):
    return NH.from_array(np.asarray(v, np.int64), name)


def model_a():
    """6 nodes, no weights, static x[2,16,8]."""
    nodes = [H.make_node("Slice", ["x", "zero", "one", "zero"], ["x_first"]),
             H.make_node("Slice", ["x", "one", "two", "zero"], ["x_second"]),
             H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
             H.make_node("ReduceMean", ["x"], ["m_x"], axes=[-1], keepdims=1),
             H.make_node("ReduceMean", ["x_swapped"], ["m_swapped"], axes=[-1], keepdims=1),
             H.make_node("Add", ["m_x", "m_swapped"], ["y"])]
    g = H.make_graph(nodes, "a", [H.make_tensor_value_info("x", TP.FLOAT, [2, 16, 8])],
                     [H.make_tensor_value_info("y", TP.FLOAT, [2, 16, 1])],
                     [i64("zero", [0]), i64("one", [1]), i64("two", [2])])
    m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
    m.ir_version = 8
    return m


def model_b(g1, b1, g2, b2):
    """y = LayerNorm1(x) + LayerNorm2(Concat(x[h:2h], x[0:h])), x[N,1024,64], halves from Shape like torch.chunk."""
    nodes = [H.make_node("Shape", ["x"], ["shape"]), H.make_node("Gather", ["shape", "zero"], ["n"], axis=0),
             H.make_node("Div", ["n", "two"], ["h"]), H.make_node("Mul", ["h", "two"], ["n2"]),
             H.make_node("Slice", ["x", "zero", "h", "zero"], ["x_first"]),
             H.make_node("Slice", ["x", "h", "n2", "zero"], ["x_second"]),
             H.make_node("Concat", ["x_second", "x_first"], ["x_swapped"], axis=0),
             H.make_node("LayerNormalization", ["x", "g1", "b1"], ["ln1"], axis=-1, epsilon=1e-6),
             H.make_node("LayerNormalization", ["x_swapped", "g2", "b2"], ["ln2"], axis=-1, epsilon=1e-6),
             H.make_node("Add", ["ln1", "ln2"], ["y"])]
    f32 = np.float32
    g = H.make_graph(nodes, "b", [H.make_tensor_value_info("x", TP.FLOAT, ["N", 1024, 64])],
                     [H.make_tensor_value_info("y", TP.FLOAT, ["N", 1024, 64])],
                     [i64("zero", [0]), i64("two", [2]), NH.from_array(g1.astype(f32), "g1"),
                      NH.from_array(b1.astype(f32), "b1"), NH.from_array(g2.astype(f32), "g2"),
                      NH.from_array(b2.astype(f32), "b2")])
    m = H.make_model(g, opset_imports=[H.make_opsetid("", 17)])
    m.ir_version = 8
    return m


def run(model, x, profile=None):
    b = trt.Builder(LOG)
    flags = 0 if RTX else (1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED) if STRONG else 0)
    net = b.create_network(flags)
    p = trt.OnnxParser(net, LOG)
    assert p.parse(model.SerializeToString()), [str(p.get_error(i)) for i in range(p.num_errors)]
    cfg = b.create_builder_config()
    if profile:
        rest = list(net.get_input(0).shape)[1:]
        pr = b.create_optimization_profile()
        pr.set_shape("x", *[[n] + rest for n in profile])
        cfg.add_optimization_profile(pr)
    eng = trt.Runtime(LOG).deserialize_cuda_engine(b.build_serialized_network(net, cfg))
    ctx = eng.create_execution_context(eng.create_runtime_config()) if RTX else eng.create_execution_context()
    xin = torch.from_numpy(x).cuda()
    ctx.set_input_shape("x", tuple(xin.shape))
    y = torch.empty(tuple(ctx.get_tensor_shape("y")), dtype=torch.float32, device="cuda")
    ctx.set_tensor_address("x", xin.data_ptr())
    ctx.set_tensor_address("y", y.data_ptr())
    s = torch.cuda.Stream()
    torch.cuda.synchronize()
    assert ctx.execute_async_v3(s.cuda_stream)
    s.synchronize()
    return y.cpu().numpy().astype(np.float64)


def err(y, ref):  # mean abs error per batch element
    return np.abs(y - ref).reshape(len(y), -1).mean(1)


def ln(v, g, b):
    mu = v.mean(-1, keepdims=True)
    return (v - mu) / np.sqrt(((v - mu) ** 2).mean(-1, keepdims=True) + 1e-6) * g + b


print(("TensorRT-RTX " if RTX else "TensorRT ") + trt.__version__, "|", torch.cuda.get_device_name(0),
      "| strongly typed FP32" if STRONG else "| FP32, no precision flags")
x = np.random.default_rng(0).standard_normal((2, 16, 8)).astype(np.float32)
xd = x.astype(np.float64)
ref = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[:1]]).mean(-1, keepdims=True)
bug = xd.mean(-1, keepdims=True) + np.concatenate([xd[1:], xd[1:]]).mean(-1, keepdims=True)
y = run(model_a(), x)
print("A  static N=2        : mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))

rng = np.random.default_rng(7)
g1, b1, g2, b2 = 0.5 + rng.random(64), 0.2 * rng.standard_normal(64), 0.5 + rng.random(64), 0.2 * rng.standard_normal(64)
mb = model_b(g1, b1, g2, b2)
x = np.random.default_rng(1).standard_normal((2, 1024, 64)).astype(np.float32)
xd = x.astype(np.float64)
ref = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[:1]]), g2, b2)
bug = ln(xd, g1, b1) + ln(np.concatenate([xd[1:], xd[1:]]), g2, b2)
for name, prof in (("static N=2  ", (2, 2, 2)), ("dynamic 2..32", (2, 8, 32))):
    y = run(mb, x, prof)
    print(f"B  {name}, run N=2: mean|y-ref| per batch element", err(y, ref), " | vs 'swapped=[x1,x1]'", err(y, bug))

Output. TensorRT 10.14.1.48.post1, TensorRT 11.4.0.106 and TensorRT-RTX 1.7.1.116 print the same numbers; only the first line differs.

TensorRT 10.14.1.48.post1 | NVIDIA GeForce RTX 5090 Laptop GPU | FP32, no precision flags
A  static N=2        : mean|y-ref| per batch element [1.88265403e-08 4.11452092e-01]  | vs 'swapped=[x1,x1]' [1.88265403e-08 1.91575964e-08]
B  static N=2  , run N=2: mean|y-ref| per batch element [7.75532025e-08 1.15770741e+00]  | vs 'swapped=[x1,x1]' [7.75532025e-08 9.34008044e-08]
B  dynamic 2..32, run N=2: mean|y-ref| per batch element [7.75532025e-08 7.76504523e-08]  | vs 'swapped=[x1,x1]' [7.75532025e-08 1.15770741e+00]
TensorRT 11.4.0.106 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
TensorRT-RTX 1.7.1.116 | NVIDIA GeForce RTX 5090 Laptop GPU | strongly typed FP32
(remaining lines identical to the above)

How to read the output:

  • Static engines (A, and B static): batch element 1 is off by 0.41 / 1.16 from the reference, but matches the "swapped = [x1, x1]" result to ~1e-7.
  • Dynamic engine (B dynamic): correct.

Have you tried the latest release?: Yes. 11.4.0.106 is the latest tensorrt-cu13 wheel, and 1.7.1.116 is the latest tensorrt-rtx wheel.

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes. ONNX Runtime CPU gives the correct result for both models (mean abs error vs the numpy reference: A 1.9e-8, B 9.9e-8). The TensorRT dynamic-shape engine is also correct.

主要语言
C++
星标
13.5k
派生
2.4k
平均合并
6 天 21 小时
30 天内合并 PR
3

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/TensorRT 的其他 Issue

查看 NVIDIA/TensorRT 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。