FP32 ReduceSum becomes very slow when used inside an encoder (TensorRT 10.8, RTX 5060 Laptop)
Les mainteneurs répondent en général sous 6 jours
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Accessibilité débutants
- 55/100
- Type d'issue
- Bug
- Clarté
- Clairement spécifiée
- Activité
- Active
- Stack technique
- cpp
- Domaine
- machine-learning, performance
Piste de recherche
Start by running the attached reproduce.py script with TensorRT 10.8 and the supplied ONNX models; compare the layer profiles for encoder-reduce and encoder-matmul, focusing on __myl_CasResDivMulSum_myl0_2. The issue provides no TensorRT source file or failing test to locate, so tracing the fused kernel and identifying the cause will require familiarity with TensorRT internals. Done means explaining the slowdown and whether a supported option avoids it, with reproducible profile evidence.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Description
I found a large speed difference in a small ONNX test model.
ReduceSum followed by division takes less than 1 ms when tested alone.
When I add a few encoder layers after it, TensorRT combines the sum and division
into one GPU kernel. This kernel takes about 359 ms.
I also tested another way to calculate the same sum: multiply the input by a
vector of ones using FP32 MatMul. With this change, the whole encoder takes
about 17 ms instead of 375 ms.
The attached script creates the models, random weights and random input data.
It does not contain trained model weights or private data.
Environment
TensorRT Version: 10.8.0.43 (trtexec: v100800, build b43)
NVIDIA GPU: NVIDIA GeForce RTX 5060 Laptop GPU, 8 GB, compute capability 12.0
NVIDIA Driver Version: 595.99.02
CUDA Version: 12.8, nvcc V12.8.93
CUDNN Version: 8.9.7 (libcudnn.so.8, version API returns 8907)
Operating System: Ubuntu 22.04.5 LTS, x86_64; kernel 7.0.0-42-generic
Python Version: 3.10.12
TensorFlow Version: Not used
PyTorch Version: Not used
ONNX Version: 1.9.0; generated model uses opset 17 and IR 7
NumPy Version: 1.23.5
Baremetal or Container: Docker container; the image name/tag is not known.
The models use fixed input shapes, strongly typed FP32, TF32 disabled and a
4096 MiB workspace limit. No other CUDA compute process was running during
the first tests. GPU clock speeds were not locked.
Relevant Files
Model link: See the attached tensorrt-reduce-repro.zip.
The ZIP contains four small ONNX models, the script that creates them,
benchmark logs, layer profiles and output checks. All weights and inputs
are synthetic. The script creates the large input files locally, so these
files and GPU-specific engine files are not included in the attachment.
Steps To Reproduce
Commands or scripts:
Install NumPy and ONNX, and use a TensorRT installation that includes trtexec.
Extract the attachment and run:
python3 reproduce.py --trtexec /path/to/trtexec --output-dir repro-output
This creates and tests four models. It saves the logs, layer details and timing
results in repro-output. The default is 60 seconds for each timing run, with
a separate profiling run. Use --duration 0 for a shorter test of at least ten
iterations per run.
To create the files without running the tests:
python3 reproduce.py --generate-only --output-dir repro-output
You can then test a model directly:
trtexec \
--onnx=repro-output/encoder-reduce.onnx \
--stronglyTyped --noTF32 --memPoolSize=workspace:4096 \
--profilingVerbosity=detailed \
--loadInputs=voxels:repro-output/voxels.bin,num_points:repro-output/num_points.bin \
--warmUp=300 --duration=0 --iterations=10 --separateProfileRun \
--exportProfile=repro-output/encoder-reduce-profile.json \
--exportLayerInfo=repro-output/encoder-reduce-layers.json
Repeat with mean-reduce, encoder-tail and encoder-matmul. Change the input
model name and output file names in the command for each test.
Have you tried the latest release? No. These measurements are from
TensorRT 10.8.0.43. I have not checked whether the latest release fixes this.
Can this model run on other frameworks? Yes, with the following test limits.
ONNX Runtime 1.12.0 with CPUExecutionProvider runs the full-size mean-only
model successfully. It also runs all three encoder versions with eight pillars.
I used eight pillars for the CPU encoder tests to limit memory use; I have not
run the full 60000-pillar encoder on ONNX Runtime. All tested outputs are finite,
and the small encoder versions agree within rtol=1e-5, atol=1e-4.
See onnxruntime-check.json and check_onnxruntime.py.
Test models
All models have these inputs:
voxels: FP32[60000,128,3]num_points: INT32[60000], with values from 1 to 128
Unused points are filled with zeros. The script uses fixed random seeds, so it
creates the same inputs and weights each time.
The main model has nine ONNX nodes:
sum = ReduceSum(voxels, axis=1, keepdims=1)
count = Reshape(Cast(num_points, FLOAT), [60000,1,1])
mean = sum / count
centered = voxels - mean
features = Concat(voxels, centered, voxels, axis=2)
encoded = MatMul(features, random_weight_9x64)
output = ReduceMax(Relu(encoded), axis=1)
The four tests are:
-
mean-reduce: stops atmean, without the encoder layers. -
encoder-reduce: uses the full model shown above. -
encoder-tail: moves the points to the last dimension before summing,
then moves the result back:Transpose(ReduceSum(Transpose(voxels, [0,2,1]), axis=2, keepdims=1), [0,2,1]) -
encoder-matmul: replaces only the sum with FP32 matrix multiplication:Transpose(MatMul(Transpose(voxels, [0,2,1]), ones_fp32[128,1]), [0,2,1])
All encoder tests use the same weights and inputs. Both forms calculate the
same mathematical sum, but floating-point rounding can produce small differences.
Results
| Model | Total GPU layer time | Slowest layer |
|---|---|---|
mean-reduce |
0.391 ms | 0.383 ms |
encoder-reduce |
374.992 ms | 359.159 ms |
encoder-tail |
377.700 ms | 362.008 ms |
encoder-matmul |
16.981 ms | 7.407 ms |
The full encoder with ReduceSum is about 22 times slower than the version
with MatMul.
The totals are the sum of the average times for all GPU layers in the separate
profiling run. They do not include the whole application's processing time.
The logs also include the normal benchmark's GPU compute times.
In both slow encoders, the layer __myl_CasResDivMulSum_myl0_2 takes most of the
time. The last-dimension rewrite still produces this slow combined kernel.
The mean-only model uses a fast separate sum layer. The MatMul encoder does not
use the slow combined sum kernel.
Longer verbose run
I also ran both main models for at least 60 seconds with --verbose,
--dumpLayerInfo and --dumpProfile, as requested in the reporting guide.
The average sum of GPU layer times was:
encoder-reduce: 377.193 msencoder-matmul: 17.305 ms
The full logs are encoder-reduce-60s-verbose.log and
encoder-matmul-60s-verbose.log. Nsight Systems traces are not included.
Output comparison
All outputs contain finite numbers.
- The last-dimension version gives exactly the same outputs as the original
version, including every bit. - The MatMul version has a maximum absolute output difference of about
5.72e-6. It passesnumpy.allclose(rtol=1e-5, atol=1e-4).
See numerical-check.json. This comparison checks the small test model only;
it does not measure the detection accuracy of a trained model.
Questions
I expected the sum and division to stay fast when used inside the encoder.
Adding the encoder layers should add their own cost, but the sum itself becomes
much slower.
- Is this a known problem when TensorRT combines these operations?
- Can I keep
ReduceSumand prevent TensorRT from creating this slow kernel? - Is there a recommended TensorRT version or build option for this case?
I have not tested other TensorRT versions yet. I do not know the exact reason
for the slow GPU kernel. The MatMul change is a possible workaround, but it
still needs accuracy checks before use with a trained model.
Related issues checked
- #2308: two fused groups
starting withReduceMeantook about 60% of inference time on TensorRT 8.4.3.1
and A100. It was closed after no further activity. It does not confirm the
same problem, but it is the closest report I found. - #1491: a user reported
addReduce()about 10 times slower than PyTorch on TensorRT 7.1.3 and Jetson
TX2. A maintainer said this should be fixed in TensorRT 8.2. Our report uses
TensorRT 10.8 and the reduction alone is fast, so this may be a different case. - #3962: slower
InstanceNormalization after moving from TensorRT 8.6 to 10. It concerns a
different operation and was closed after no further activity.
I did not find an existing issue with this exact nine-node reproduction or
the __myl_CasResDivMulSum kernel name.
- Langage dominant
- C++
- Étoiles
- 13.5k
- Forks
- 2.4k
- Merge moyen
- 6 j 21 h
- PR mergées (30 j)
- 3
Préparer son environnement
- Aucun Dockerfile ni fichier Docker Compose
- Aucun modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de NVIDIA/TensorRT
-
Module:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 77/100
Les mainteneurs répondent en général sous 6 jours
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyPeut-être pris @0z5a l’a pris il y a 16 jours. OuverteModule:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 6 jours
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerancePeut-être pris @0z5a l’a pris il y a 16 jours. OuverteModule:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 6 jours
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodePeut-être pris @gabortar l’a pris il y a 39 jours. OuverteModule:GraphSurgeon
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
Les mainteneurs répondent en général sous 6 jours
-
Polygraphy parse_meta accepts a shape with a missing bracketPeut-être pris @SashaMIT l’a pris il y a 51 jours. OuverteModule:Polygraphy
Difficulté 2/5 1-3 heures Accessibilité débutants 86/100
Les mainteneurs répondent en général sous 6 jours
Toutes les issues de NVIDIA/TensorRT
Issues similaires
-
Difficulté 1/5 Moins d'une heure Accessibilité débutants 91/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 69/100
maplibre/maplibre-native#5072 ·
Les mainteneurs répondent en général sous 1 jour
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 79/100
tenstorrent/tt-umd#4017 ·
Les mainteneurs répondent en général sous 3 jours
-
difficulty:easy func:gui prio:low type:bug
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
NatronGitHub/Natron#1538 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
hyprwm/hyprsunset#103 · 1 commentaire ·