Builder keeps one host copy of a weight for every layer that reads it transposed
Los mantenedores suelen responder en 6 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 25/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Área
- compilers, performance
Línea de trabajo
Start by running python trt_transpose_readers.py one_constant 10 and python trt_transpose_readers.py pretransposed 10 to reproduce the gap in host peak memory. The payload names no source files, so the builder code that handles transposed matrix-multiply weights has to be located. Done means one transposed constant read by N layers keeps host peak close to the pretransposed numbers, about 6.9 GiB for N=10.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
What is wrong
When several matrix multiplies read the same weight with MatrixOperation.TRANSPOSE, the builder seems to keep a separate host copy of that weight for each reader. In the script below, one 1 GiB float32 weight read by 10 layers needs 16.1 GiB of host memory to build. Transposed once up front and then read by the same 10 layers, it needs 6.9 GiB. Each extra transposed reader adds about 1 GiB, one more copy of the weight.
Who hits it: any model that runs one block many times in one graph. Examples are a flow-matching or diffusion policy with its denoising loop unrolled, or classifier-free guidance that runs the same layers twice. Linear layers read their weight transposed, so this is the common case. Through Torch-TensorRT, a 3.5 GiB float32 policy that runs its action transformer 10 times needed 20.5 GiB of host memory to compile, and 11.0 GiB once the converter shared one transposed copy itself (pytorch/TensorRT#4965). Related: pytorch/TensorRT#4867.
What this blocks
xvla and eo1 (pytorch/executorch#24353) read the same weights in many layers, so their compile keeps one extra host copy per reader.
Torch-TensorRT 2.15 avoids it only with the opt-in deduplicate_weights (pytorch/TensorRT#4965). Anyone who builds with the TensorRT API, or without that option, still pays it.
A fix in the builder keeps that memory flat however many layers read a weight, which matters most on Jetsons with little memory.
Repro
python trt_transpose_readers.py MODE [N] [D], with MODE one of:
per_reader: N constant layers that point at the same host memory, each read transposed.one_constant: one constant layer, read transposed by N layers.pretransposed: one constant, transposed once with a shuffle layer, read by N layers without a transpose flag.
Default N = 10, D = 16384 (1 GiB float32). TF32 is off.
| mode | N = 1 | N = 5 | N = 10 |
|---|---|---|---|
| per_reader | 16061 MiB (15.7x) | ||
| one_constant | 6918 MiB (6.8x) | 10939 MiB (10.7x) | 16060 MiB (15.7x) |
| pretransposed | 6931 MiB (6.8x) | 6931 MiB (6.8x) | 6930 MiB (6.8x) |
Host peak during the build, weight 1024 MiB, host memory before the build 1603 MiB. The plan is 1024 MiB in every case.
Expected
Host memory that does not grow with the number of readers, as in pretransposed: at most one builder copy of the weight, however many layers read it transposed.
Versions
TensorRT 11.4.0.106, numpy 2.5.4, H100, driver 580.126.09, Linux x86_64.
trt_transpose_readers.py
import sys, threading, time
import numpy as np
import tensorrt as trt
MODE = sys.argv[1]
N = int(sys.argv[2]) if len(sys.argv) > 2 else 10
D = int(sys.argv[3]) if len(sys.argv) > 3 else 16384
peak = [0]
def rss_mib():
for line in open("/proc/self/status"):
if line.startswith("VmRSS"):
return int(line.split()[1]) // 1024
def sampler():
while True:
peak[0] = max(peak[0], rss_mib())
time.sleep(0.05)
w = (np.random.default_rng(0).standard_normal((D, D)) / np.sqrt(D)).astype(np.float32)
w_mib = w.nbytes >> 20
builder = trt.Builder(trt.Logger(trt.Logger.WARNING))
net = builder.create_network(0)
x = net.add_input("x", trt.float32, (8, D))
def constant():
return net.add_constant((D, D), trt.Weights(w)).get_output(0)
shared = constant() if MODE in ("one_constant", "pretransposed") else None
if MODE == "pretransposed":
t = net.add_shuffle(shared)
t.first_transpose = (1, 0)
shared = t.get_output(0)
op = trt.MatrixOperation.NONE if MODE == "pretransposed" else trt.MatrixOperation.TRANSPOSE
for _ in range(N):
weight = shared if shared is not None else constant()
y = net.add_matrix_multiply(x, trt.MatrixOperation.NONE, weight, op).get_output(0)
x = net.add_activation(y, trt.ActivationType.TANH).get_output(0)
net.mark_output(x)
cfg = builder.create_builder_config()
cfg.clear_flag(trt.BuilderFlag.TF32)
start = rss_mib()
threading.Thread(target=sampler, daemon=True).start()
plan = builder.build_serialized_network(net, cfg)
time.sleep(0.2)
print(f"{MODE}: N={N}, weight {w_mib} MiB, host before build {start} MiB, "
f"build peak {peak[0]} MiB ({peak[0] / w_mib:.1f}x the weight), plan {plan.nbytes >> 20} MiB")
Actual
per_reader: N=10, weight 1024 MiB, host before build 1603 MiB, build peak 16061 MiB (15.7x the weight), plan 1024 MiB
one_constant: N=10, weight 1024 MiB, host before build 1603 MiB, build peak 16060 MiB (15.7x the weight), plan 1024 MiB
one_constant: N=5, weight 1024 MiB, host before build 1603 MiB, build peak 10939 MiB (10.7x the weight), plan 1024 MiB
one_constant: N=1, weight 1024 MiB, host before build 1603 MiB, build peak 6918 MiB (6.8x the weight), plan 1024 MiB
pretransposed: N=10, weight 1024 MiB, host before build 1603 MiB, build peak 6930 MiB (6.8x the weight), plan 1024 MiB
pretransposed: N=5, weight 1024 MiB, host before build 1603 MiB, build peak 6931 MiB (6.8x the weight), plan 1024 MiB
pretransposed: N=1, weight 1024 MiB, host before build 1603 MiB, build peak 6931 MiB (6.8x the weight), plan 1024 MiB
- Lenguaje dominante
- C++
- Estrellas
- 13.5k
- Forks
- 2.4k
- Merge medio
- 6 d 21 h
- PR fusionados (30 d)
- 3
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de NVIDIA/TensorRT
-
Module:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 6 días
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyPosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerancePosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodePosiblemente ocupada @gabortar la tomó hace 39 días. AbiertoModule:GraphSurgeon
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 6 días
-
Polygraphy parse_meta accepts a shape with a missing bracketPosiblemente ocupada @SashaMIT la tomó hace 51 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
Los mantenedores suelen responder en 6 días
Todos los issues de NVIDIA/TensorRT
Issues similares
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 79/100
tenstorrent/tt-umd#4017 ·
Los mantenedores suelen responder en 3 días
-
difficulty:easy func:gui prio:low type:bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
NatronGitHub/Natron#1538 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
hyprwm/hyprsunset#103 · 1 comentario ·
-
Feature request
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
qbittorrent/qBittorrent#25338 ·
Los mantenedores suelen responder en 2 días
-
Needs: Attention Needs: Version Info Platform: Android Platform: tvOS
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
react/react-native#60180 · 3 comentarios ·
Los mantenedores suelen responder en 2 días