FP32 ReduceSum becomes very slow when used inside an encoder (TensorRT 10.8, RTX 5060 Laptop)
メンテナーはふだん 6 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 55/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- cpp
調査の方向性
Start by running the attached reproduce.py script with TensorRT 10.8 and the supplied ONNX models; compare the layer profiles for encoder-reduce and encoder-matmul, focusing on __myl_CasResDivMulSum_myl0_2. The issue provides no TensorRT source file or failing test to locate, so tracing the fused kernel and identifying the cause will require familiarity with TensorRT internals. Done means explaining the slowdown and whether a supported option avoids it, with reproducible profile evidence.
索引モデルが issue の本文から書いたものです。
説明
Description
I found a large speed difference in a small ONNX test model.
ReduceSum followed by division takes less than 1 ms when tested alone.
When I add a few encoder layers after it, TensorRT combines the sum and division
into one GPU kernel. This kernel takes about 359 ms.
I also tested another way to calculate the same sum: multiply the input by a
vector of ones using FP32 MatMul. With this change, the whole encoder takes
about 17 ms instead of 375 ms.
The attached script creates the models, random weights and random input data.
It does not contain trained model weights or private data.
Environment
TensorRT Version: 10.8.0.43 (trtexec: v100800, build b43)
NVIDIA GPU: NVIDIA GeForce RTX 5060 Laptop GPU, 8 GB, compute capability 12.0
NVIDIA Driver Version: 595.99.02
CUDA Version: 12.8, nvcc V12.8.93
CUDNN Version: 8.9.7 (libcudnn.so.8, version API returns 8907)
Operating System: Ubuntu 22.04.5 LTS, x86_64; kernel 7.0.0-42-generic
Python Version: 3.10.12
TensorFlow Version: Not used
PyTorch Version: Not used
ONNX Version: 1.9.0; generated model uses opset 17 and IR 7
NumPy Version: 1.23.5
Baremetal or Container: Docker container; the image name/tag is not known.
The models use fixed input shapes, strongly typed FP32, TF32 disabled and a
4096 MiB workspace limit. No other CUDA compute process was running during
the first tests. GPU clock speeds were not locked.
Relevant Files
Model link: See the attached tensorrt-reduce-repro.zip.
The ZIP contains four small ONNX models, the script that creates them,
benchmark logs, layer profiles and output checks. All weights and inputs
are synthetic. The script creates the large input files locally, so these
files and GPU-specific engine files are not included in the attachment.
Steps To Reproduce
Commands or scripts:
Install NumPy and ONNX, and use a TensorRT installation that includes trtexec.
Extract the attachment and run:
python3 reproduce.py --trtexec /path/to/trtexec --output-dir repro-output
This creates and tests four models. It saves the logs, layer details and timing
results in repro-output. The default is 60 seconds for each timing run, with
a separate profiling run. Use --duration 0 for a shorter test of at least ten
iterations per run.
To create the files without running the tests:
python3 reproduce.py --generate-only --output-dir repro-output
You can then test a model directly:
trtexec \
--onnx=repro-output/encoder-reduce.onnx \
--stronglyTyped --noTF32 --memPoolSize=workspace:4096 \
--profilingVerbosity=detailed \
--loadInputs=voxels:repro-output/voxels.bin,num_points:repro-output/num_points.bin \
--warmUp=300 --duration=0 --iterations=10 --separateProfileRun \
--exportProfile=repro-output/encoder-reduce-profile.json \
--exportLayerInfo=repro-output/encoder-reduce-layers.json
Repeat with mean-reduce, encoder-tail and encoder-matmul. Change the input
model name and output file names in the command for each test.
Have you tried the latest release? No. These measurements are from
TensorRT 10.8.0.43. I have not checked whether the latest release fixes this.
Can this model run on other frameworks? Yes, with the following test limits.
ONNX Runtime 1.12.0 with CPUExecutionProvider runs the full-size mean-only
model successfully. It also runs all three encoder versions with eight pillars.
I used eight pillars for the CPU encoder tests to limit memory use; I have not
run the full 60000-pillar encoder on ONNX Runtime. All tested outputs are finite,
and the small encoder versions agree within rtol=1e-5, atol=1e-4.
See onnxruntime-check.json and check_onnxruntime.py.
Test models
All models have these inputs:
voxels: FP32[60000,128,3]num_points: INT32[60000], with values from 1 to 128
Unused points are filled with zeros. The script uses fixed random seeds, so it
creates the same inputs and weights each time.
The main model has nine ONNX nodes:
sum = ReduceSum(voxels, axis=1, keepdims=1)
count = Reshape(Cast(num_points, FLOAT), [60000,1,1])
mean = sum / count
centered = voxels - mean
features = Concat(voxels, centered, voxels, axis=2)
encoded = MatMul(features, random_weight_9x64)
output = ReduceMax(Relu(encoded), axis=1)
The four tests are:
-
mean-reduce: stops atmean, without the encoder layers. -
encoder-reduce: uses the full model shown above. -
encoder-tail: moves the points to the last dimension before summing,
then moves the result back:Transpose(ReduceSum(Transpose(voxels, [0,2,1]), axis=2, keepdims=1), [0,2,1]) -
encoder-matmul: replaces only the sum with FP32 matrix multiplication:Transpose(MatMul(Transpose(voxels, [0,2,1]), ones_fp32[128,1]), [0,2,1])
All encoder tests use the same weights and inputs. Both forms calculate the
same mathematical sum, but floating-point rounding can produce small differences.
Results
| Model | Total GPU layer time | Slowest layer |
|---|---|---|
mean-reduce |
0.391 ms | 0.383 ms |
encoder-reduce |
374.992 ms | 359.159 ms |
encoder-tail |
377.700 ms | 362.008 ms |
encoder-matmul |
16.981 ms | 7.407 ms |
The full encoder with ReduceSum is about 22 times slower than the version
with MatMul.
The totals are the sum of the average times for all GPU layers in the separate
profiling run. They do not include the whole application's processing time.
The logs also include the normal benchmark's GPU compute times.
In both slow encoders, the layer __myl_CasResDivMulSum_myl0_2 takes most of the
time. The last-dimension rewrite still produces this slow combined kernel.
The mean-only model uses a fast separate sum layer. The MatMul encoder does not
use the slow combined sum kernel.
Longer verbose run
I also ran both main models for at least 60 seconds with --verbose,
--dumpLayerInfo and --dumpProfile, as requested in the reporting guide.
The average sum of GPU layer times was:
encoder-reduce: 377.193 msencoder-matmul: 17.305 ms
The full logs are encoder-reduce-60s-verbose.log and
encoder-matmul-60s-verbose.log. Nsight Systems traces are not included.
Output comparison
All outputs contain finite numbers.
- The last-dimension version gives exactly the same outputs as the original
version, including every bit. - The MatMul version has a maximum absolute output difference of about
5.72e-6. It passesnumpy.allclose(rtol=1e-5, atol=1e-4).
See numerical-check.json. This comparison checks the small test model only;
it does not measure the detection accuracy of a trained model.
Questions
I expected the sum and division to stay fast when used inside the encoder.
Adding the encoder layers should add their own cost, but the sum itself becomes
much slower.
- Is this a known problem when TensorRT combines these operations?
- Can I keep
ReduceSumand prevent TensorRT from creating this slow kernel? - Is there a recommended TensorRT version or build option for this case?
I have not tested other TensorRT versions yet. I do not know the exact reason
for the slow GPU kernel. The MatMul change is a possible workaround, but it
still needs accuracy checks before use with a trained model.
Related issues checked
- #2308: two fused groups
starting withReduceMeantook about 60% of inference time on TensorRT 8.4.3.1
and A100. It was closed after no further activity. It does not confirm the
same problem, but it is the closest report I found. - #1491: a user reported
addReduce()about 10 times slower than PyTorch on TensorRT 7.1.3 and Jetson
TX2. A maintainer said this should be fixed in TensorRT 8.2. Our report uses
TensorRT 10.8 and the reduction alone is fast, so this may be a different case. - #3962: slower
InstanceNormalization after moving from TensorRT 8.6 to 10. It concerns a
different operation and was closed after no further activity.
I did not find an existing issue with this exact nine-node reproduction or
the __myl_CasResDivMulSum kernel name.
- 主要言語
- C++
- スター
- 13.5k
- フォーク
- 2.4k
- 平均マージ
- 6日 21時間
- マージ済み PR(30日)
- 3
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
NVIDIA/TensorRT のほかの issue
-
Module:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
メンテナーはふだん 6 日以内に返信
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectly対応中かも @0z5a が 16 日前に担当しました。 オープンModule:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 6 日以内に返信
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerance対応中かも @0z5a が 16 日前に担当しました。 オープンModule:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 6 日以内に返信
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop node対応中かも @gabortar が 39 日前に担当しました。 オープンModule:GraphSurgeon
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 6 日以内に返信
-
Polygraphy parse_meta accepts a shape with a missing bracket対応中かも @SashaMIT が 51 日前に担当しました。 オープンModule:Polygraphy
難易度 2/5 1〜3時間 初心者へのやさしさ 86/100
メンテナーはふだん 6 日以内に返信
NVIDIA/TensorRT の issue をすべて見る
似ている issue
-
難易度 1/5 1時間未満 初心者へのやさしさ 91/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 69/100
maplibre/maplibre-native#5072 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 79/100
tenstorrent/tt-umd#4017 ·
メンテナーはふだん 3 日以内に返信
-
difficulty:easy func:gui prio:low type:bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
NatronGitHub/Natron#1538 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
hyprwm/hyprsunset#103 · コメント 1 件 ·