AOT plugins leak CUDA modules on dynamic-shape changes in TensorRT 11.1/11.2.1
Maintainer antworten meist innerhalb von 6 Tagen
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Anfängerfreundlichkeit
- 35/100
- Issue-Typ
- Bug
- Klarheit
- Klar beschrieben
- Aktivitätsstatus
- Aktiv
- Bereich
- ai-infra-agents, performance
Rechercherichtung
Beginne damit, den angehängten Reproducer zu extrahieren und run.sh sowie replay.cc zu lesen, und führe dann die dokumentierten Kontrollen für feste Shapes und sich ändernde Shapes aus. Verwende das gemeldete Verhalten beim Laden von Modulen und beim Teardown, um den Lebenszyklus des AOT-Plugins nachzuverfolgen; abgeschlossen ist die Aufgabe, wenn wiederholte zulässige Shape-Änderungen einen begrenzten Speicherverbrauch haben und der TensorRT-Teardown geladene Module freigibt, ohne die erfolgreichen Ausgabekontrollen zu beeinträchtigen.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Description
GPU memory accumulates when an engine containing an AOT plugin is executed repeatedly with different input shapes in the same execution context. The extra memory remains after destroying the execution context, engine, and TensorRT runtime. Returning to previously used shapes continues the growth.
The attached minimal reproducer uses one hand-written PTX kernel implementing output[i] = 2 * input[i] + 1, registered through tensorrt.plugin.aot_impl. Inference runs in a standalone C++ TensorRT/CUDA process.
The same kernel and one tactic are used for all shapes. The plugin has no allocation or caching code. All outputs match an independent CPU reference exactly. The standard TensorRT affine and convolution controls do not show the retained memory observed with the AOT plugin.
Observed behavior
The clean-environment TensorRT 11.2.1.2 run produces the following results. Each row runs 30,000 requests in a separate process. Memory differences are measured by cudaMemGetInfo between a fresh CUDA context and destruction of all TensorRT objects, without a profiler:
| Engine | Input H/W | Shape changes | Extra memory after TensorRT teardown |
|---|---|---|---|
Standard Scale (2*x + 1) |
Fixed 128x128 | 0 | 0 bytes |
Standard Scale (2*x + 1) |
W alternates 128/192 | 19,999 | 0 bytes |
| Standard 3x3 convolution | Fixed 128x128 | 0 | 0 bytes |
| Standard 3x3 convolution | W alternates 128/192 | 19,999 | 0 bytes |
| One AOT affine plugin | Fixed 128x128 | 0 | 0 bytes |
| One AOT affine plugin | W alternates 128/192 | 19,999 | 50,331,648 bytes (48 MiB) |
The AOT fixed/width pair and standard Scale width control produce the same memory differences on TensorRT 11.1.0.106. These byte counts describe this particular kernel and environment; the issue is continued accumulation, not a particular allocation size or threshold.
A separate TensorRT 11.2.1.2 CUPTI trace records 20,000 successful cuModuleLoadData calls: the first inference plus 19,999 shape changes. Every loaded module's requested kernel is affine. Teardown issues one successful cuModuleUnload, for the last loaded module. The earlier 19,999 modules remain loaded. Module-load caller stacks enter libnvinfer.so.11.
Additional controls reproduce retention when only H changes, or when H/W are transposed between 128x192 and 192x128 with unchanged element count. Three chained AOT plugins retain three modules per shape change. Skipping redundant setInputShape calls does not eliminate retention.
The CUDA context stays alive during the TensorRT-teardown measurements. Separately destroying it with cudaDeviceReset removes the extra memory: both fixed-shape and changing-shape controls reach the same 4 MiB device reading while their processes remain alive, then the 1 MiB idle reading on process exit. Destroying TensorRT execution contexts alone does not have this effect.
Expected behavior
Repeatedly revisiting the same legal input shapes should have bounded resource usage. CUDA modules loaded for AOT execution should be released when they are no longer needed, including when their owning TensorRT objects are destroyed. A long-running process should not accumulate GPU memory with each shape change.
Environment
TensorRT Version: 11.2.1.2; also reproduced on 11.1.0.106. Each engine was built with the same TensorRT version used to execute it. Official, unmodified tensorrt-cu13 Python bindings/libraries.
NVIDIA GPU: RTX 3080 Ti, 12 GiB, compute capability 8.6. One otherwise idle GPU was selected for every experiment.
NVIDIA Driver Version: 580.173.02.
CUDA Version: CUDA toolkit 13.0; nvcc 13.0.88.
CUDNN Version: Not installed in the clean reproducer environment and not required by its source or link command.
Operating System: Ubuntu 24.04.5 LTS, Linux x86_64.
Python Version: 3.13.15, used for engine construction only. NumPy 2.5.1.
C++ compiler: GCC/G++ 13.3.0, C++17. Consumer headers are TensorRT 11.2.1.2; the 11.1 controls use the same shared API subset with their matching 11.1 engine/runtime. The clean 11.2.1 reproduction uses matching headers throughout.
Relevant Files
Attached tensorrt_shape_memory.tar.gz for reproducer.
Steps To Reproduce
Commands or scripts:
Extract tensorrt_shape_memory.tar.gz. Use a CUDA 13 development installation and G++ on Linux. Install NumPy and the matching TensorRT Python packages in an isolated Python 3.13 environment:
python3.13 -m venv .venv
.venv/bin/python -m pip install numpy==2.5.1 tensorrt-cu13==11.2.1.2
Obtain the matching public TensorRT 11.2.1.2 headers:
git clone https://github.com/NVIDIA/TensorRT.git TensorRT-headers
git -C TensorRT-headers checkout c93b7d4893184af4882f9f1862e13a5c64b8677d
Run the attached script on one idle GPU (the reported tests selected GPU 0):
GPU=0 \
PYTHON="$PWD/.venv/bin/python" \
TRT_INCLUDE="$PWD/TensorRT-headers/include" \
TRT_LIB_DIR="$PWD/.venv/lib/python3.13/site-packages/tensorrt_libs" \
CUDA_ROOT=/usr/local/cuda \
REQUESTS=30000 \
bash run.sh
The script compiles the C++ consumer, builds each engine in a separate process, and runs each fixed/width control in another fresh process. Full verbose logs are saved in the output directory it reports.
All engines use input [1,1,H,W], one optimization profile with minimum [1,1,64,64], optimum [1,1,128,128], and maximum [1,1,1152,1152]. The width workload repeats 128x128, 128x128, 128x192. Input/output buffers are allocated once at maximum test size, every request is synchronized, and every output is checked against the CPU reference. The aliased-plugin-I/O preview feature is not enabled.
The affected result reports extra_bytes_after_runtime_destruction: 50331648 with passed: true; all five controls report zero extra bytes. There is no inference exception or numerical failure; GPU memory and module lifetimes are the failing behavior.
Have you tried the latest release?: Tested 11.1.0.106 and 11.2.1.2. TensorRT 11.3.0 has not been tested.
Can this model run on other frameworks?: This is a TensorRT network-API reproducer, not an exported framework model. All outputs match the independent CPU implementations in replay.cc, and the equivalent standard TensorRT Scale control executes without the observed retention.
- Vorherrschende Sprache
- C++
- Sterne
- 13.5k
- Forks
- 2.4k
- Ø Merge
- 6 T. 21 Std.
- Gemergte PRs (30 T.)
- 3
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Keine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus NVIDIA/TensorRT
-
Module:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 77/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyEvtl. vergeben @0z5a hat das vor 16 Tagen übernommen. OffenModule:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Polygraphy INT32 comparison can overflow and falsely pass at zero toleranceEvtl. vergeben @0z5a hat das vor 16 Tagen übernommen. OffenModule:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 6 Tagen
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodeEvtl. vergeben @gabortar hat das vor 39 Tagen übernommen. OffenModule:GraphSurgeon
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
Maintainer antworten meist innerhalb von 6 Tagen
-
Polygraphy parse_meta accepts a shape with a missing bracketEvtl. vergeben @SashaMIT hat das vor 51 Tagen übernommen. OffenModule:Polygraphy
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 86/100
Maintainer antworten meist innerhalb von 6 Tagen
Alle Issues in NVIDIA/TensorRT
Ähnliche Issues
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 91/100
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 69/100
maplibre/maplibre-native#5072 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 79/100
tenstorrent/tt-umd#4017 ·
Maintainer antworten meist innerhalb von 3 Tagen
-
difficulty:easy func:gui prio:low type:bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
NatronGitHub/Natron#1538 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
hyprwm/hyprsunset#103 · 1 Kommentar ·