FP32 ReduceSum becomes very slow when used inside an encoder (TensorRT 10.8, RTX 5060 Laptop)
I maintainer di solito rispondono entro 6 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 55/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- cpp
- Ambito
- machine-learning, performance
Direzione di ricerca
Start by running the attached reproduce.py script with TensorRT 10.8 and the supplied ONNX models; compare the layer profiles for encoder-reduce and encoder-matmul, focusing on __myl_CasResDivMulSum_myl0_2. The issue provides no TensorRT source file or failing test to locate, so tracing the fused kernel and identifying the cause will require familiarity with TensorRT internals. Done means explaining the slowdown and whether a supported option avoids it, with reproducible profile evidence.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Description
I found a large speed difference in a small ONNX test model.
ReduceSum followed by division takes less than 1 ms when tested alone.
When I add a few encoder layers after it, TensorRT combines the sum and division
into one GPU kernel. This kernel takes about 359 ms.
I also tested another way to calculate the same sum: multiply the input by a
vector of ones using FP32 MatMul. With this change, the whole encoder takes
about 17 ms instead of 375 ms.
The attached script creates the models, random weights and random input data.
It does not contain trained model weights or private data.
Environment
TensorRT Version: 10.8.0.43 (trtexec: v100800, build b43)
NVIDIA GPU: NVIDIA GeForce RTX 5060 Laptop GPU, 8 GB, compute capability 12.0
NVIDIA Driver Version: 595.99.02
CUDA Version: 12.8, nvcc V12.8.93
CUDNN Version: 8.9.7 (libcudnn.so.8, version API returns 8907)
Operating System: Ubuntu 22.04.5 LTS, x86_64; kernel 7.0.0-42-generic
Python Version: 3.10.12
TensorFlow Version: Not used
PyTorch Version: Not used
ONNX Version: 1.9.0; generated model uses opset 17 and IR 7
NumPy Version: 1.23.5
Baremetal or Container: Docker container; the image name/tag is not known.
The models use fixed input shapes, strongly typed FP32, TF32 disabled and a
4096 MiB workspace limit. No other CUDA compute process was running during
the first tests. GPU clock speeds were not locked.
Relevant Files
Model link: See the attached tensorrt-reduce-repro.zip.
The ZIP contains four small ONNX models, the script that creates them,
benchmark logs, layer profiles and output checks. All weights and inputs
are synthetic. The script creates the large input files locally, so these
files and GPU-specific engine files are not included in the attachment.
Steps To Reproduce
Commands or scripts:
Install NumPy and ONNX, and use a TensorRT installation that includes trtexec.
Extract the attachment and run:
python3 reproduce.py --trtexec /path/to/trtexec --output-dir repro-output
This creates and tests four models. It saves the logs, layer details and timing
results in repro-output. The default is 60 seconds for each timing run, with
a separate profiling run. Use --duration 0 for a shorter test of at least ten
iterations per run.
To create the files without running the tests:
python3 reproduce.py --generate-only --output-dir repro-output
You can then test a model directly:
trtexec \
--onnx=repro-output/encoder-reduce.onnx \
--stronglyTyped --noTF32 --memPoolSize=workspace:4096 \
--profilingVerbosity=detailed \
--loadInputs=voxels:repro-output/voxels.bin,num_points:repro-output/num_points.bin \
--warmUp=300 --duration=0 --iterations=10 --separateProfileRun \
--exportProfile=repro-output/encoder-reduce-profile.json \
--exportLayerInfo=repro-output/encoder-reduce-layers.json
Repeat with mean-reduce, encoder-tail and encoder-matmul. Change the input
model name and output file names in the command for each test.
Have you tried the latest release? No. These measurements are from
TensorRT 10.8.0.43. I have not checked whether the latest release fixes this.
Can this model run on other frameworks? Yes, with the following test limits.
ONNX Runtime 1.12.0 with CPUExecutionProvider runs the full-size mean-only
model successfully. It also runs all three encoder versions with eight pillars.
I used eight pillars for the CPU encoder tests to limit memory use; I have not
run the full 60000-pillar encoder on ONNX Runtime. All tested outputs are finite,
and the small encoder versions agree within rtol=1e-5, atol=1e-4.
See onnxruntime-check.json and check_onnxruntime.py.
Test models
All models have these inputs:
voxels: FP32[60000,128,3]num_points: INT32[60000], with values from 1 to 128
Unused points are filled with zeros. The script uses fixed random seeds, so it
creates the same inputs and weights each time.
The main model has nine ONNX nodes:
sum = ReduceSum(voxels, axis=1, keepdims=1)
count = Reshape(Cast(num_points, FLOAT), [60000,1,1])
mean = sum / count
centered = voxels - mean
features = Concat(voxels, centered, voxels, axis=2)
encoded = MatMul(features, random_weight_9x64)
output = ReduceMax(Relu(encoded), axis=1)
The four tests are:
-
mean-reduce: stops atmean, without the encoder layers. -
encoder-reduce: uses the full model shown above. -
encoder-tail: moves the points to the last dimension before summing,
then moves the result back:Transpose(ReduceSum(Transpose(voxels, [0,2,1]), axis=2, keepdims=1), [0,2,1]) -
encoder-matmul: replaces only the sum with FP32 matrix multiplication:Transpose(MatMul(Transpose(voxels, [0,2,1]), ones_fp32[128,1]), [0,2,1])
All encoder tests use the same weights and inputs. Both forms calculate the
same mathematical sum, but floating-point rounding can produce small differences.
Results
| Model | Total GPU layer time | Slowest layer |
|---|---|---|
mean-reduce |
0.391 ms | 0.383 ms |
encoder-reduce |
374.992 ms | 359.159 ms |
encoder-tail |
377.700 ms | 362.008 ms |
encoder-matmul |
16.981 ms | 7.407 ms |
The full encoder with ReduceSum is about 22 times slower than the version
with MatMul.
The totals are the sum of the average times for all GPU layers in the separate
profiling run. They do not include the whole application's processing time.
The logs also include the normal benchmark's GPU compute times.
In both slow encoders, the layer __myl_CasResDivMulSum_myl0_2 takes most of the
time. The last-dimension rewrite still produces this slow combined kernel.
The mean-only model uses a fast separate sum layer. The MatMul encoder does not
use the slow combined sum kernel.
Longer verbose run
I also ran both main models for at least 60 seconds with --verbose,
--dumpLayerInfo and --dumpProfile, as requested in the reporting guide.
The average sum of GPU layer times was:
encoder-reduce: 377.193 msencoder-matmul: 17.305 ms
The full logs are encoder-reduce-60s-verbose.log and
encoder-matmul-60s-verbose.log. Nsight Systems traces are not included.
Output comparison
All outputs contain finite numbers.
- The last-dimension version gives exactly the same outputs as the original
version, including every bit. - The MatMul version has a maximum absolute output difference of about
5.72e-6. It passesnumpy.allclose(rtol=1e-5, atol=1e-4).
See numerical-check.json. This comparison checks the small test model only;
it does not measure the detection accuracy of a trained model.
Questions
I expected the sum and division to stay fast when used inside the encoder.
Adding the encoder layers should add their own cost, but the sum itself becomes
much slower.
- Is this a known problem when TensorRT combines these operations?
- Can I keep
ReduceSumand prevent TensorRT from creating this slow kernel? - Is there a recommended TensorRT version or build option for this case?
I have not tested other TensorRT versions yet. I do not know the exact reason
for the slow GPU kernel. The MatMul change is a possible workaround, but it
still needs accuracy checks before use with a trained model.
Related issues checked
- #2308: two fused groups
starting withReduceMeantook about 60% of inference time on TensorRT 8.4.3.1
and A100. It was closed after no further activity. It does not confirm the
same problem, but it is the closest report I found. - #1491: a user reported
addReduce()about 10 times slower than PyTorch on TensorRT 7.1.3 and Jetson
TX2. A maintainer said this should be fixed in TensorRT 8.2. Our report uses
TensorRT 10.8 and the reduction alone is fast, so this may be a different case. - #3962: slower
InstanceNormalization after moving from TensorRT 8.6 to 10. It concerns a
different operation and was closed after no further activity.
I did not find an existing issue with this exact nine-node reproduction or
the __myl_CasResDivMulSum kernel name.
- Lingua principale
- C++
- Stelle
- 13.5k
- Fork
- 2.4k
- Merge medio
- 6g 21h
- PR unite (30g)
- 3
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di NVIDIA/TensorRT
-
Module:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyForse già presa @0z5a l’ha presa 16 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy INT32 comparison can overflow and falsely pass at zero toleranceForse già presa @0z5a l’ha presa 16 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 6 giorni
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodeForse già presa @gabortar l’ha presa 39 giorni fa. ApertaModule:GraphSurgeon
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy parse_meta accepts a shape with a missing bracketForse già presa @SashaMIT l’ha presa 51 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
I maintainer di solito rispondono entro 6 giorni
Tutte le issue di NVIDIA/TensorRT
Issue simili
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 79/100
tenstorrent/tt-umd#4017 ·
I maintainer di solito rispondono entro 3 giorni
-
difficulty:easy func:gui prio:low type:bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
NatronGitHub/Natron#1538 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
hyprwm/hyprsunset#103 · 1 commento ·
-
Feature request
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
qbittorrent/qBittorrent#25338 ·
I maintainer di solito rispondono entro 2 giorni
-
Needs: Attention Needs: Version Info Platform: Android Platform: tvOS
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
react/react-native#60180 · 3 commenti ·
I maintainer di solito rispondono entro 2 giorni