float32 attention on inputs cast from bf16 crashes the builder or fails to build (casts folded into IAttention)
Los mantenedores suelen responder en 6 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 6/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Área
- machine-learning
Línea de trabajo
The fix sits in TensorRT's closed-source Myelin attention path, so the repo's open components are unlikely to hold it. Start by running the repro script in the issue for one failing case (for example float,0,bf16) to confirm the segfault or the element-type error. Done means every case in the results table builds in float32 or the casts are kept, and the repro table shows no CRASHED or BUILD FAILED rows.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
What is wrong
Some models compute attention in float32 on purpose for accuracy: they cast bf16 (or fp16) query, key and value to
float32, run attention, and keep or cast the result. TensorRT 11.4 folds those casts into its attention layer. It
then builds only if the whole layer can run in one 16-bit type: no float32 mask, and the output cast back to 16 bits.
In the other cases the build crashes (segmentation fault), or rejects the float32 mask and reports that no implementation was found. The same network with float32 network inputs (no casts) builds.
So a model that casts its bf16 query, key and value to float32 for a more accurate attention cannot be built.
What this blocks
fastwam (pytorch/executorch#24353) computes attention this way. It builds today only because Torch-TensorRT 2.15 splits such attention into plain matmul, softmax and matmul (pytorch/TensorRT#4964), which gives up the fused kernel.
A fix lets fastwam and similar models use the fused attention kernel again.
Repro
Needs tensorrt. Each case builds one strongly typed network in its own process (some cases crash the process) and reports whether it builds.
import argparse, subprocess, sys
import numpy as np
import tensorrt as trt
p = argparse.ArgumentParser()
p.add_argument("--shape", default="8,8,50,805,64", help="B,H,L,S,D")
p.add_argument("--case", help="MASK,CAST_BACK: run one case per process, because some cases crash")
a = p.parse_args()
B, H, L, S, D = (int(x) for x in a.shape.split(","))
logger = trt.Logger(trt.Logger.WARNING)
def build(mask_kind, cast_back, input_type=trt.bfloat16):
builder = trt.Builder(logger)
net = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
q = net.add_input("q", input_type, (B, H, L, D))
k = net.add_input("k", input_type, (B, H, S, D))
v = net.add_input("v", input_type, (B, H, S, D))
if input_type != trt.float32:
q, k, v = (net.add_cast(t, trt.float32).get_output(0) for t in (q, k, v))
att = net.add_attention_v2(q, k, v, trt.AttentionNormalizationOp.SOFTMAX, trt.CausalMaskKind.NONE)
att.decomposable = True
if mask_kind == "float":
att.mask = net.add_input("mask", trt.float32, (B, 1, 1, S))
elif mask_kind == "bool":
att.mask = net.add_input("mask", trt.bool, (B, 1, 1, S))
out = att.get_output(0)
if cast_back:
out = net.add_cast(out, trt.bfloat16).get_output(0)
net.mark_output(out)
try:
plan = builder.build_serialized_network(net, builder.create_builder_config())
return "builds" if plan is not None else "BUILD FAILED"
except Exception as error:
return f"BUILD FAILED: {error}"
if a.case:
mask_kind, cast_back, inputs = a.case.split(",")
print(build(mask_kind, cast_back == "1", trt.float32 if inputs == "fp32" else trt.bfloat16), flush=True)
sys.exit(0)
print("TensorRT", trt.__version__, "shape", (B, H, L, S, D), flush=True)
cases = [(m, c, "bf16") for m in ("none", "bool", "float") for c in (True, False)]
cases += [(m, False, "fp32") for m in ("none", "bool", "float")] # control: float32 inputs, no casts to fold
for mask_kind, cast_back, inputs in cases:
r = subprocess.run([sys.executable, __file__, "--shape", a.shape, "--case", f"{mask_kind},{int(cast_back)},{inputs}"],
capture_output=True, text=True)
text = (r.stdout + r.stderr).splitlines()
out = [l for l in r.stdout.splitlines() if l.startswith(("builds", "BUILD"))]
errors = [l.strip() for l in text if "differs from" in l][:1]
errors += [l.strip()[l.find("Error Code"):] for l in text if "[E]" in l and "Error Code" in l][-1:]
result = out[-1] if r.returncode == 0 and out else f"CRASHED (exit code {r.returncode})"
if result != "builds" and errors:
result += " | " + " | ".join(e[:200] for e in errors)
print(f"inputs={inputs} mask={mask_kind:5} output={'bf16' if cast_back else 'fp32'}: {result}", flush=True)
Results
| inputs | mask | output | H100 (sm_90) | A100 (sm_80) |
|---|---|---|---|---|
| bf16, cast to fp32 | none | bf16 | builds | builds |
| bf16, cast to fp32 | none | fp32 | segfault | segfault |
| bf16, cast to fp32 | bool | bf16 | builds | builds |
| bf16, cast to fp32 | bool | fp32 | segfault | segfault |
| bf16, cast to fp32 | float (fp32) | bf16 | fails: Myelin "Input 0's element type (bf16) differs from input 1's element type (float)", then "Could not find any implementation" | fails, same error |
| bf16, cast to fp32 | float (fp32) | fp32 | segfault | segfault |
| fp32 (control) | none, bool, float | fp32 | builds | builds |
Actual
H100, driver 580.126.09 (A100 prints the same 9 lines):
TensorRT 11.4.0.106 shape (8, 8, 50, 805, 64)
inputs=bf16 mask=none output=bf16: builds
inputs=bf16 mask=none output=fp32: CRASHED (exit code -11)
inputs=bf16 mask=bool output=bf16: builds
inputs=bf16 mask=bool output=fp32: CRASHED (exit code -11)
inputs=bf16 mask=float output=bf16: BUILD FAILED | Input 0's element type (bf16) differs from input 1's element type (float).; [myelin_graph.h:1295: attachExceptionMsgToGraph] MyelinCheckException: operation.cpp:1570: CHECK_OK(infer_shape_t::canonical | Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[(Unnamed Layer* 2) [Cast]...(Unnamed Layer* 6) [Cast]]}. In computeCosts at /_src/optimizer/common/tactic/optimiz
inputs=bf16 mask=float output=fp32: CRASHED (exit code -11)
inputs=fp32 mask=none output=fp32: builds
inputs=fp32 mask=bool output=fp32: builds
inputs=fp32 mask=float output=fp32: builds
Versions: TensorRT 11.4.0.106 (pip), Linux x86_64, H100 and A100 80 GB, driver 580.126.09.
Expected
Every case builds and runs in float32, or TensorRT keeps the casts when it cannot fold them.
What we tried
Torch-TensorRT now decomposes these cases into float32 matmul, softmax and matmul. That works and matches PyTorch,
but it gives up the fused kernel.
Possibly related: #4842 (bf16 strongly typed build segfaults in Myelin).
- Lenguaje dominante
- C++
- Estrellas
- 13.5k
- Forks
- 2.4k
- Merge medio
- 6 d 21 h
- PR fusionados (30 d)
- 3
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de NVIDIA/TensorRT
-
Module:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 6 días
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyPosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerancePosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodePosiblemente ocupada @gabortar la tomó hace 39 días. AbiertoModule:GraphSurgeon
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 6 días
-
Polygraphy parse_meta accepts a shape with a missing bracketPosiblemente ocupada @SashaMIT la tomó hace 51 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
Los mantenedores suelen responder en 6 días
Todos los issues de NVIDIA/TensorRT
Issues similares
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 79/100
tenstorrent/tt-umd#4017 ·
Los mantenedores suelen responder en 3 días
-
difficulty:easy func:gui prio:low type:bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
NatronGitHub/Natron#1538 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
hyprwm/hyprsunset#103 · 1 comentario ·
-
Feature request
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
qbittorrent/qBittorrent#25338 ·
Los mantenedores suelen responder en 2 días
-
Needs: Attention Needs: Version Info Platform: Android Platform: tvOS
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
react/react-native#60180 · 3 comentarios ·
Los mantenedores suelen responder en 2 días