float32 attention on inputs cast from bf16 crashes the builder or fails to build (casts folded into IAttention)

Đang mở
#4,936 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 6 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
6/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp, python
Lĩnh vực
machine-learning

Hướng nghiên cứu

The fix sits in TensorRT's closed-source Myelin attention path, so the repo's open components are unlikely to hold it. Start by running the repro script in the issue for one failing case (for example float,0,bf16) to confirm the segfault or the element-type error. Done means every case in the results table builds in float32 or the casts are kept, and the repro table shows no CRASHED or BUILD FAILED rows.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Module:Engine Build

What is wrong

Some models compute attention in float32 on purpose for accuracy: they cast bf16 (or fp16) query, key and value to
float32, run attention, and keep or cast the result. TensorRT 11.4 folds those casts into its attention layer. It
then builds only if the whole layer can run in one 16-bit type: no float32 mask, and the output cast back to 16 bits.
In the other cases the build crashes (segmentation fault), or rejects the float32 mask and reports that no implementation was found. The same network with float32 network inputs (no casts) builds.

So a model that casts its bf16 query, key and value to float32 for a more accurate attention cannot be built.

What this blocks

fastwam (pytorch/executorch#24353) computes attention this way. It builds today only because Torch-TensorRT 2.15 splits such attention into plain matmul, softmax and matmul (pytorch/TensorRT#4964), which gives up the fused kernel.
A fix lets fastwam and similar models use the fused attention kernel again.

Repro

Needs tensorrt. Each case builds one strongly typed network in its own process (some cases crash the process) and reports whether it builds.

import argparse, subprocess, sys
import numpy as np
import tensorrt as trt

p = argparse.ArgumentParser()
p.add_argument("--shape", default="8,8,50,805,64", help="B,H,L,S,D")
p.add_argument("--case", help="MASK,CAST_BACK: run one case per process, because some cases crash")
a = p.parse_args()
B, H, L, S, D = (int(x) for x in a.shape.split(","))
logger = trt.Logger(trt.Logger.WARNING)


def build(mask_kind, cast_back, input_type=trt.bfloat16):
    builder = trt.Builder(logger)
    net = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
    q = net.add_input("q", input_type, (B, H, L, D))
    k = net.add_input("k", input_type, (B, H, S, D))
    v = net.add_input("v", input_type, (B, H, S, D))
    if input_type != trt.float32:
        q, k, v = (net.add_cast(t, trt.float32).get_output(0) for t in (q, k, v))
    att = net.add_attention_v2(q, k, v, trt.AttentionNormalizationOp.SOFTMAX, trt.CausalMaskKind.NONE)
    att.decomposable = True
    if mask_kind == "float":
        att.mask = net.add_input("mask", trt.float32, (B, 1, 1, S))
    elif mask_kind == "bool":
        att.mask = net.add_input("mask", trt.bool, (B, 1, 1, S))
    out = att.get_output(0)
    if cast_back:
        out = net.add_cast(out, trt.bfloat16).get_output(0)
    net.mark_output(out)
    try:
        plan = builder.build_serialized_network(net, builder.create_builder_config())
        return "builds" if plan is not None else "BUILD FAILED"
    except Exception as error:
        return f"BUILD FAILED: {error}"


if a.case:
    mask_kind, cast_back, inputs = a.case.split(",")
    print(build(mask_kind, cast_back == "1", trt.float32 if inputs == "fp32" else trt.bfloat16), flush=True)
    sys.exit(0)
print("TensorRT", trt.__version__, "shape", (B, H, L, S, D), flush=True)
cases = [(m, c, "bf16") for m in ("none", "bool", "float") for c in (True, False)]
cases += [(m, False, "fp32") for m in ("none", "bool", "float")]  # control: float32 inputs, no casts to fold
for mask_kind, cast_back, inputs in cases:
    r = subprocess.run([sys.executable, __file__, "--shape", a.shape, "--case", f"{mask_kind},{int(cast_back)},{inputs}"],
                       capture_output=True, text=True)
    text = (r.stdout + r.stderr).splitlines()
    out = [l for l in r.stdout.splitlines() if l.startswith(("builds", "BUILD"))]
    errors = [l.strip() for l in text if "differs from" in l][:1]
    errors += [l.strip()[l.find("Error Code"):] for l in text if "[E]" in l and "Error Code" in l][-1:]
    result = out[-1] if r.returncode == 0 and out else f"CRASHED (exit code {r.returncode})"
    if result != "builds" and errors:
        result += " | " + " | ".join(e[:200] for e in errors)
    print(f"inputs={inputs} mask={mask_kind:5} output={'bf16' if cast_back else 'fp32'}: {result}", flush=True)

Results

inputs mask output H100 (sm_90) A100 (sm_80)
bf16, cast to fp32 none bf16 builds builds
bf16, cast to fp32 none fp32 segfault segfault
bf16, cast to fp32 bool bf16 builds builds
bf16, cast to fp32 bool fp32 segfault segfault
bf16, cast to fp32 float (fp32) bf16 fails: Myelin "Input 0's element type (bf16) differs from input 1's element type (float)", then "Could not find any implementation" fails, same error
bf16, cast to fp32 float (fp32) fp32 segfault segfault
fp32 (control) none, bool, float fp32 builds builds

Actual

H100, driver 580.126.09 (A100 prints the same 9 lines):

TensorRT 11.4.0.106 shape (8, 8, 50, 805, 64)
inputs=bf16 mask=none  output=bf16: builds
inputs=bf16 mask=none  output=fp32: CRASHED (exit code -11)
inputs=bf16 mask=bool  output=bf16: builds
inputs=bf16 mask=bool  output=fp32: CRASHED (exit code -11)
inputs=bf16 mask=float output=bf16: BUILD FAILED | Input 0's element type (bf16) differs from input 1's element type (float).; [myelin_graph.h:1295: attachExceptionMsgToGraph] MyelinCheckException: operation.cpp:1570: CHECK_OK(infer_shape_t::canonical | Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[(Unnamed Layer* 2) [Cast]...(Unnamed Layer* 6) [Cast]]}. In computeCosts at /_src/optimizer/common/tactic/optimiz
inputs=bf16 mask=float output=fp32: CRASHED (exit code -11)
inputs=fp32 mask=none  output=fp32: builds
inputs=fp32 mask=bool  output=fp32: builds
inputs=fp32 mask=float output=fp32: builds

Versions: TensorRT 11.4.0.106 (pip), Linux x86_64, H100 and A100 80 GB, driver 580.126.09.

Expected

Every case builds and runs in float32, or TensorRT keeps the casts when it cannot fold them.

What we tried

Torch-TensorRT now decomposes these cases into float32 matmul, softmax and matmul. That works and matches PyTorch,
but it gives up the fused kernel.

Possibly related: #4842 (bf16 strongly typed build segfaults in Myelin).

Ngôn ngữ chính
C++
Star
13.5k
Fork
2.4k
Merge trung bình
6 ngày 21 giờ
Pull request đã merge (30 ngày)
3

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/TensorRT

Tất cả issue của NVIDIA/TensorRT

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.