Serialized engine size failure of TensorRT 11.2 when building an unfused pointwise op on a large activation on GPU H100
維護者通常 6 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 新手友好度
- 55/100
- Issue 類型
- 缺陷
- 描述清晰度
- 描述清楚
- 活躍度
- 活躍
研究方向
從提供的 trt_repro.py 腳本開始,比較 TensorRT 11.2.1.2 和 10.16.1.11 中的序列化 plan。追蹤 build_serialized_network,尋找回退到 Myelin 的未融合 pointwise 操作,並以回報的 _myl 節點和 builder_optimization_level 的行為作為線索。完成的標準是修正此 regression,且不要求重新排序 graph,同時包含涵蓋 plan 大小的 regression test。
由索引模型根據 Issue 內容生成。
描述
Description
TensorRT 11.2.1.2 appends a large zero-filled region to the serialized plan for any
pointwise op that cannot be fused into an adjacent Conv. The appended region is
exactly 7x the fp16 output activation tensor, so the plan size scales with activation
geometry rather than with weights or graph size.
A single-node LeakyRelu graph on a [1,64,2944,1664] fp16 tensor serializes to
4,389,349,804 bytes (99.998% zeros) on TensorRT 11.2.1.2, versus 10,508 bytes on
TensorRT 10.16.1.11 — a 417,714x increase for the same ONNX on the same GPU, driver,
container and build script.
The engine is functionally correct: it deserializes and produces bit-identical output. The
problem is purely the serialized artifact size.
The same LeakyRelu fused into a Conv produces a 210,812-byte plan that is
byte-identical on 10.16.1.11 and 11.2.1.2. Only fusability differs between the working
and failing case — the activation shape is the same.
We first hit this on a real model (RealBasicVSR x4 upsample head), where a DepthToSpace
sat between the Conv and its LeakyReLU and blocked the fusion twice. That engine built
at 5,487,774,180 bytes under 11.2.1.2 versus 1,041,804 bytes under 10.16.1.11
(5268x). Byte layout of the 5.2 GB file: real content occupies the leading 1,100,136 bytes,
followed by a single contiguous zero run of 5,486,674,044 bytes to EOF. That equals
exactly 7x the two unfused activations combined:
7 * 2 * (1*64*1472*832) = 1,097,334,784
7 * 2 * (1*64*2944*1664) = 4,389,339,136
-------------
5,486,673,920 (vs 5,486,674,044 observed)
Environment
TensorRT Version: 11.2.1.2 (regression vs 10.16.1.11, which is unaffected)
NVIDIA GPU: NVIDIA H100 80GB HBM3
NVIDIA Driver Version: 570.172.08
CUDA Version: 12.8.1
CUDNN Version: 91002
Operating System: Ubuntu (Linux 6.8.0-1037-gcp x86_64, glibc 2.39)
Python Version (if applicable): 3.12.3
Tensorflow Version (if applicable): n/a
PyTorch Version (if applicable): 2.9.1+cu128 (used only to cross-check numerics; not needed for the repro)
Baremetal or Container (if so, version): Container, based on nvidia/cuda:12.8.1 with TensorRT 11.2.1.2 installed
Relevant Files
Model link: none needed — the repro below builds the graph programmatically in ~20 lines
and requires no model file, no weights, and no external data.
Steps To Reproduce
Commands or scripts:
Save as trt_repro.py and run python trt_repro.py. Requires only tensorrt, onnx, numpy.
"""Minimal repro: TensorRT 11.2 emits a huge zero-filled serialized plan for an
unfused pointwise op.
TensorRT 10.16.1.11 -> 10,508 bytes
TensorRT 11.2.1.2 -> 4,389,349,804 bytes (99.998% zeros)
"""
import numpy as np
import onnx
import tensorrt as trt
from onnx import TensorProto as T
from onnx import helper, numpy_helper
SHAPE = [1, 64, 2944, 1664] # fp16 activation; 313,524,224 elements
def build(nodes, initializers, path):
x = helper.make_tensor_value_info("x", T.FLOAT16, SHAPE)
y = helper.make_tensor_value_info("y", T.FLOAT16, SHAPE)
graph = helper.make_graph(nodes, "repro", [x], [y], initializers)
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 20)])
model.ir_version = 10
onnx.save(model, path)
logger = trt.Logger(trt.Logger.ERROR)
builder = trt.Builder(logger)
network = builder.create_network(
1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
)
parser = trt.OnnxParser(network, logger)
with open(path, "rb") as f:
assert parser.parse(f.read()), [
parser.get_error(i) for i in range(parser.num_errors)
]
config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 4 << 30)
profile = builder.create_optimization_profile()
profile.set_shape("x", SHAPE, SHAPE, SHAPE)
config.add_optimization_profile(profile)
return bytes(builder.build_serialized_network(network, config))
def report(label, plan):
elements = 1
for d in SHAPE:
elements *= d
print(
f"{label:34s} {len(plan):>14,} bytes "
f"zeros={100 * plan.count(0) / len(plan):6.3f}% "
f"plan/(fp16 tensor)={len(plan) / (elements * 2):.3f}"
)
print(f"TensorRT {trt.__version__}\n")
# 1. A single unfused LeakyRelu -> plan is ~7x the output activation, all zeros.
report(
"LeakyRelu alone (BUG)",
build([helper.make_node("LeakyRelu", ["x"], ["y"], alpha=0.1)], [], "/tmp/a.onnx"),
)
# 2. The same LeakyRelu fused into a Conv -> normal plan size.
# Only the fusability differs; the activation shape is identical.
w = numpy_helper.from_array(np.zeros((64, 64, 3, 3), dtype=np.float16), "w")
b = numpy_helper.from_array(np.zeros((64,), dtype=np.float16), "b")
report(
"Conv+LeakyRelu (fuses, OK)",
build(
[
helper.make_node(
"Conv", ["x", "w", "b"], ["t"],
pads=[1, 1, 1, 1], strides=[1, 1], dilations=[1, 1], group=1,
),
helper.make_node("LeakyRelu", ["t"], ["y"], alpha=0.1),
],
[w, b],
"/tmp/b.onnx",
),
)
Actual output (same machine, same GPU, same script; only the container's TensorRT differs):
########## TensorRT 11.2.1.2 ##########
TensorRT 11.2.1.2
LeakyRelu alone (BUG) 4,389,349,804 bytes zeros=100.000% plan/(fp16 tensor)=7.000
Conv+LeakyRelu (fuses, OK) 210,812 bytes zeros=62.273% plan/(fp16 tensor)=0.000
########## TensorRT 10.16.1.11 ##########
TensorRT 10.16.1.11
LeakyRelu alone (BUG) 10,508 bytes zeros=48.677% plan/(fp16 tensor)=0.000
Conv+LeakyRelu (fuses, OK) 210,812 bytes zeros=62.273% plan/(fp16 tensor)=0.000
Note the Conv+LeakyRelu plan is byte-identical (210,812) across both versions, while
the standalone LeakyRelu differs by 417,714x.
The engine inspector shows the unfused op falling to Myelin
On the real model, the two affected activations appear as Myelin nodes:
__myl_CastMulLtSeleCast_myl6_0
__myl_CastMulLtSeleCast_myl13_0
TensorRT 10.16.1.11 produces the same 17 layers with the same two __myl_ nodes at
1,041,804 bytes, so the layer structure is not what changed between versions — only the
amount of zero padding written into the plan.
Not specific to LeakyRelu
An exact Mul+Max rewrite of the same LeakyRelu bloats identically
(4,389,350,020 bytes, 100% zeros), so the trigger is an unfused pointwise op in general,
not this operator.
No builder configuration avoids it
All of the following produce a byte-identical 4,389,349,804-byte plan on 11.2.1.2:
| setting | plan size |
|---|---|
| default | 4,389,349,804 |
tiling_optimization_level = NONE |
4,389,349,804 |
l2_limit_for_tiling = 0 |
4,389,349,804 |
max_num_tactics = 1 |
4,389,349,804 |
BuilderFlag.STRIP_PLAN |
4,389,349,804 |
BuilderFlag.DISABLE_COMPILATION_CACHE |
4,389,349,804 |
builder_optimization_level = 1..5 |
4,389,349,804 |
builder_optimization_level = 0 |
10,532 (clean) |
STRIP_PLAN having no effect confirms the region is not weight data.
builder_optimization_level = 0 is the only clean setting, but it disables tactic
selection and is not usable for a production engine.
EXCLUDE_LEAN_RUNTIME is rejected with
kEXCLUDE_LEAN_RUNTIME can only be set if kVERSION_COMPATIBLE is set, which confirms
VERSION_COMPATIBLE is off and no lean runtime is being embedded.
Workaround
Reordering the graph so the pointwise op can fuse into its Conv avoids the issue entirely.
In our case DepthToSpace is a pure position permutation and LeakyReLU is elementwise,
so they commute exactly; emitting Conv -> LeakyReLU -> DepthToSpace instead of
Conv -> DepthToSpace -> LeakyReLU gave:
- 5,487,774,180 bytes -> 1,056,260 bytes
- zero
__myl_nodes; all activations fused asconv + PWN(leaky_relu) - bit-identical output (
torch.equalTrue) against both the bloated 11.2 engine and a
10.16.1.11 engine built from the same ONNX
This workaround is only available when the graph happens to admit a commuting reorder, so
it is not a general fix.
Have you tried the latest release?:
Yes — 11.2.1.2 is the version exhibiting the issue. 10.16.1.11 is unaffected.
Can this model run on other frameworks?:
Yes. The graph is a single standard ONNX LeakyRelu node (opset 20) and runs correctly
under ONNXRuntime. The TensorRT engine also executes correctly — the defect is the size of
the serialized plan, not its numerics.
Impact
Engines are distributed to inference workers by download. A 1 MB artifact becoming 5.2 GB
inflates storage, transfer, and cold-start time for every worker, for a graph whose real
content is ~1 MB. Because the padding scales with activation size, it is worst exactly
where it hurts most — high-resolution upsampling heads.
- 主要語言
- C++
- 星號
- 13.5k
- 分支
- 2.4k
- 平均合併
- 6 天 21 小時
- 30 天內合併 PR
- 3
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
NVIDIA/TensorRT 的其他 Issue
-
Module:Polygraphy
難度 2/5 1-3 小時 新手友好度 77/100
維護者通常 6 天內回覆
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectly可能已有人在做 @0z5a 於 16 天前認領。 未關閉Module:Polygraphy
難度 2/5 1-3 小時 新手友好度 72/100
維護者通常 6 天內回覆
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerance可能已有人在做 @0z5a 於 16 天前認領。 未關閉Module:Polygraphy
難度 2/5 1-3 小時 新手友好度 72/100
維護者通常 6 天內回覆
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop node可能已有人在做 @gabortar 於 39 天前認領。 未關閉Module:GraphSurgeon
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 6 天內回覆
-
Polygraphy parse_meta accepts a shape with a missing bracket可能已有人在做 @SashaMIT 於 51 天前認領。 未關閉Module:Polygraphy
難度 2/5 1-3 小時 新手友好度 86/100
維護者通常 6 天內回覆
相似的 Issue
-
難度 1/5 1 小時以內 新手友好度 91/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 69/100
maplibre/maplibre-native#5072 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 79/100
tenstorrent/tt-umd#4017 ·
維護者通常 3 天內回覆
-
difficulty:easy func:gui prio:low type:bug
難度 2/5 1-3 小時 新手友好度 72/100
NatronGitHub/Natron#1538 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 72/100
hyprwm/hyprsunset#103 · 1 則留言 ·