float32 attention on inputs cast from bf16 crashes the builder or fails to build (casts folded into IAttention)
I maintainer di solito rispondono entro 6 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 6/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Ambito
- machine-learning
Direzione di ricerca
The fix sits in TensorRT's closed-source Myelin attention path, so the repo's open components are unlikely to hold it. Start by running the repro script in the issue for one failing case (for example float,0,bf16) to confirm the segfault or the element-type error. Done means every case in the results table builds in float32 or the casts are kept, and the repro table shows no CRASHED or BUILD FAILED rows.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
What is wrong
Some models compute attention in float32 on purpose for accuracy: they cast bf16 (or fp16) query, key and value to
float32, run attention, and keep or cast the result. TensorRT 11.4 folds those casts into its attention layer. It
then builds only if the whole layer can run in one 16-bit type: no float32 mask, and the output cast back to 16 bits.
In the other cases the build crashes (segmentation fault), or rejects the float32 mask and reports that no implementation was found. The same network with float32 network inputs (no casts) builds.
So a model that casts its bf16 query, key and value to float32 for a more accurate attention cannot be built.
What this blocks
fastwam (pytorch/executorch#24353) computes attention this way. It builds today only because Torch-TensorRT 2.15 splits such attention into plain matmul, softmax and matmul (pytorch/TensorRT#4964), which gives up the fused kernel.
A fix lets fastwam and similar models use the fused attention kernel again.
Repro
Needs tensorrt. Each case builds one strongly typed network in its own process (some cases crash the process) and reports whether it builds.
import argparse, subprocess, sys
import numpy as np
import tensorrt as trt
p = argparse.ArgumentParser()
p.add_argument("--shape", default="8,8,50,805,64", help="B,H,L,S,D")
p.add_argument("--case", help="MASK,CAST_BACK: run one case per process, because some cases crash")
a = p.parse_args()
B, H, L, S, D = (int(x) for x in a.shape.split(","))
logger = trt.Logger(trt.Logger.WARNING)
def build(mask_kind, cast_back, input_type=trt.bfloat16):
builder = trt.Builder(logger)
net = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
q = net.add_input("q", input_type, (B, H, L, D))
k = net.add_input("k", input_type, (B, H, S, D))
v = net.add_input("v", input_type, (B, H, S, D))
if input_type != trt.float32:
q, k, v = (net.add_cast(t, trt.float32).get_output(0) for t in (q, k, v))
att = net.add_attention_v2(q, k, v, trt.AttentionNormalizationOp.SOFTMAX, trt.CausalMaskKind.NONE)
att.decomposable = True
if mask_kind == "float":
att.mask = net.add_input("mask", trt.float32, (B, 1, 1, S))
elif mask_kind == "bool":
att.mask = net.add_input("mask", trt.bool, (B, 1, 1, S))
out = att.get_output(0)
if cast_back:
out = net.add_cast(out, trt.bfloat16).get_output(0)
net.mark_output(out)
try:
plan = builder.build_serialized_network(net, builder.create_builder_config())
return "builds" if plan is not None else "BUILD FAILED"
except Exception as error:
return f"BUILD FAILED: {error}"
if a.case:
mask_kind, cast_back, inputs = a.case.split(",")
print(build(mask_kind, cast_back == "1", trt.float32 if inputs == "fp32" else trt.bfloat16), flush=True)
sys.exit(0)
print("TensorRT", trt.__version__, "shape", (B, H, L, S, D), flush=True)
cases = [(m, c, "bf16") for m in ("none", "bool", "float") for c in (True, False)]
cases += [(m, False, "fp32") for m in ("none", "bool", "float")] # control: float32 inputs, no casts to fold
for mask_kind, cast_back, inputs in cases:
r = subprocess.run([sys.executable, __file__, "--shape", a.shape, "--case", f"{mask_kind},{int(cast_back)},{inputs}"],
capture_output=True, text=True)
text = (r.stdout + r.stderr).splitlines()
out = [l for l in r.stdout.splitlines() if l.startswith(("builds", "BUILD"))]
errors = [l.strip() for l in text if "differs from" in l][:1]
errors += [l.strip()[l.find("Error Code"):] for l in text if "[E]" in l and "Error Code" in l][-1:]
result = out[-1] if r.returncode == 0 and out else f"CRASHED (exit code {r.returncode})"
if result != "builds" and errors:
result += " | " + " | ".join(e[:200] for e in errors)
print(f"inputs={inputs} mask={mask_kind:5} output={'bf16' if cast_back else 'fp32'}: {result}", flush=True)
Results
| inputs | mask | output | H100 (sm_90) | A100 (sm_80) |
|---|---|---|---|---|
| bf16, cast to fp32 | none | bf16 | builds | builds |
| bf16, cast to fp32 | none | fp32 | segfault | segfault |
| bf16, cast to fp32 | bool | bf16 | builds | builds |
| bf16, cast to fp32 | bool | fp32 | segfault | segfault |
| bf16, cast to fp32 | float (fp32) | bf16 | fails: Myelin "Input 0's element type (bf16) differs from input 1's element type (float)", then "Could not find any implementation" | fails, same error |
| bf16, cast to fp32 | float (fp32) | fp32 | segfault | segfault |
| fp32 (control) | none, bool, float | fp32 | builds | builds |
Actual
H100, driver 580.126.09 (A100 prints the same 9 lines):
TensorRT 11.4.0.106 shape (8, 8, 50, 805, 64)
inputs=bf16 mask=none output=bf16: builds
inputs=bf16 mask=none output=fp32: CRASHED (exit code -11)
inputs=bf16 mask=bool output=bf16: builds
inputs=bf16 mask=bool output=fp32: CRASHED (exit code -11)
inputs=bf16 mask=float output=bf16: BUILD FAILED | Input 0's element type (bf16) differs from input 1's element type (float).; [myelin_graph.h:1295: attachExceptionMsgToGraph] MyelinCheckException: operation.cpp:1570: CHECK_OK(infer_shape_t::canonical | Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[(Unnamed Layer* 2) [Cast]...(Unnamed Layer* 6) [Cast]]}. In computeCosts at /_src/optimizer/common/tactic/optimiz
inputs=bf16 mask=float output=fp32: CRASHED (exit code -11)
inputs=fp32 mask=none output=fp32: builds
inputs=fp32 mask=bool output=fp32: builds
inputs=fp32 mask=float output=fp32: builds
Versions: TensorRT 11.4.0.106 (pip), Linux x86_64, H100 and A100 80 GB, driver 580.126.09.
Expected
Every case builds and runs in float32, or TensorRT keeps the casts when it cannot fold them.
What we tried
Torch-TensorRT now decomposes these cases into float32 matmul, softmax and matmul. That works and matches PyTorch,
but it gives up the fused kernel.
Possibly related: #4842 (bf16 strongly typed build segfaults in Myelin).
- Lingua principale
- C++
- Stelle
- 13.5k
- Fork
- 2.4k
- Merge medio
- 6g 21h
- PR unite (30g)
- 3
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di NVIDIA/TensorRT
-
Module:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyForse già presa @0z5a l’ha presa 16 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy INT32 comparison can overflow and falsely pass at zero toleranceForse già presa @0z5a l’ha presa 16 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 6 giorni
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodeForse già presa @gabortar l’ha presa 39 giorni fa. ApertaModule:GraphSurgeon
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 6 giorni
-
Polygraphy parse_meta accepts a shape with a missing bracketForse già presa @SashaMIT l’ha presa 51 giorni fa. ApertaModule:Polygraphy
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
I maintainer di solito rispondono entro 6 giorni
Tutte le issue di NVIDIA/TensorRT
Issue simili
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 91/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 69/100
maplibre/maplibre-native#5072 ·
I maintainer di solito rispondono entro 1 giorno
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 79/100
tenstorrent/tt-umd#4017 ·
I maintainer di solito rispondono entro 3 giorni
-
difficulty:easy func:gui prio:low type:bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
NatronGitHub/Natron#1538 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
hyprwm/hyprsunset#103 · 1 commento ·