bf16 decode attention picks `_gemv_mha_v1`, 5x to 8x slower than `_gemv_mha_v2`, and `_gemv_mha_v2` is wrong with a bool mask

Đang mở
#4,947 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 6 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
28/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Lĩnh vực
backend, performance

Hướng nghiên cứu

The issue provides a TensorRT Python API repro and names _gemv_mha_v1, _gemv_mha_v2, and _gemm_mha_v2, but no implementation files or tests. Start by running the repro on the stated TensorRT and A100 setup to confirm the timing and mask errors. Done means selecting a fast split path for long caches, handling bool masks correctly, and adding a split path for a few queries; this requires kernel and dispatch investigation.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Module:Performance

What is wrong

In decoding, attention has 1 query over a long key/value (KV) cache. For a bfloat16 IAttention layer on A100, the builder picks _gemv_mha_v1. It launches one thread block per query head (16 blocks on a GPU with 108 multiprocessors). At 4096 slots it takes 168 us per layer for 16.8 MB of keys and values, about 100 GB/s.

TensorRT also has _gemv_mha_v2, which splits the cache across many blocks (256 at 4096 slots, 512 at 8192). When I force it with the build route -mha_fusion:v1=off, it is 5x to 8x faster. But it gives correct results only when the valid length comes from key_value_lengths. With a bool mask the result is wrong (error 0.35), which fits the mask being ignored. In float16 TensorRT picks this kernel by default, and there the mask is ignored (pytorch/TensorRT#5011, measured on H100 above 512 keys). Passing key_value_lengths without the route makes the default kernel 1.8x slower.

1 query, 16 heads, 8 KV heads, head size 128 4096 slots 8192 slots
bool mask (default) 168 us, correct 324 us, correct
key_value_lengths (default) 296 us, correct 577 us, correct
key_value_lengths, -mha_fusion:v1=off 35 us, correct 43 us, correct
bool mask, -mha_fusion:v1=off 37 us, wrong 45 us, wrong

For reference, FlashAttention's split-KV kernel (flash_attn_with_kvcache, flash-attn 2.8.3, same GPU and valid length) takes 23 us and 35 us on the same shapes. With 64 queries, _gemm_mha_v2 also launches 16 blocks, and the layer takes 140 to 159 us (including 2 copies of K and V), against 28 us. No build route I tried gave it a split path.

At 4096 slots TensorRT 11.5 picks the same default kernels with the same times. With the route and a bool mask it falls back to a slower kernel (165 us) and is correct.

I found this running Alpamayo 2 Edge through Torch-TensorRT (pytorch/executorch#25253). There, my measurements put the whole gap to PyTorch with FlashAttention on A100 in attention for a few queries over the cache, in decoding and in the action head.

Expected

  • For 1 query over a long cache, pick the split kernel by default.
  • _gemv_mha_v2 either applies a bool mask or is never picked with one.
  • A split path for a few queries (tens) over a long cache.

Repro

TensorRT Python API only. 28 attention layers in a chain, so they cannot overlap.

repro.py
"""bf16 attention over a long KV cache with the TensorRT Python API (no other framework).

28 attention layers in a chain (each output is the next query), 16 query heads, 8 KV heads, head size 128.
The cache has S slots and the first 7/8 of them are valid. Each case gives the valid part either as a bool
mask [1, 1, 1, S] or as key_value_lengths. Times are per layer; error is relative to PyTorch on the same inputs.
"""
import math

import numpy as np
import tensorrt as trt
import torch
import torch.nn.functional as F

H, KVH, D, LAYERS = 16, 8, 128, 28
logger = trt.Logger(trt.Logger.WARNING)


def build(T, S, use_mask, use_lengths, route):
    builder = trt.Builder(logger)
    net = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
    x = net.add_input("q", trt.bfloat16, (1, H, T, D))
    caches = [net.add_input(f"c{i}", trt.bfloat16, (1, KVH, S, D)) for i in range(2 * LAYERS)]
    mask = net.add_input("mask", trt.bool, (1, 1, 1, S)) if use_mask else None
    lengths = net.add_input("lengths", trt.int32, (1,)) if use_lengths else None
    scale = net.add_constant((1, 1, 1, 1), trt.Weights(np.array([1 / math.sqrt(D)], dtype=np.float32))).get_output(0)
    scale = net.add_cast(scale, trt.bfloat16).get_output(0)
    for i in range(LAYERS):
        q = net.add_elementwise(x, scale, trt.ElementWiseOperation.PROD).get_output(0)
        att = net.add_attention_v2(q, caches[2 * i], caches[2 * i + 1], trt.AttentionNormalizationOp.SOFTMAX, trt.CausalMaskKind.NONE)
        if mask is not None:
            att.mask = mask
        if lengths is not None:
            att.key_value_lengths = lengths
        x = att.get_output(0)
    net.mark_output(x)
    config = builder.create_builder_config()
    if route:
        config.build_route = route
    engine = trt.Runtime(logger).deserialize_cuda_engine(builder.build_serialized_network(net, config))
    return engine, engine.create_execution_context()


def top_kernel(fn):
    with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA]) as prof:
        fn()
        torch.cuda.synchronize()
    k = max((e for e in prof.key_averages() if e.device_time_total > 0), key=lambda e: e.device_time_total)
    return k.key.split("_0x")[0]


def us_per_layer(fn, iters=200):
    for _ in range(10):
        fn()
    torch.cuda.synchronize()
    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    s.record()
    for _ in range(iters):
        fn()
    e.record()
    torch.cuda.synchronize()
    return s.elapsed_time(e) / iters / LAYERS * 1000


print(trt.__version__, torch.cuda.get_device_name())
torch.cuda.set_stream(torch.cuda.Stream())  # TensorRT warns about the default stream
torch.manual_seed(0)
for T, S in ((1, 4096), (1, 8192), (64, 4096)):
    valid = S - S // 8
    q = torch.randn(1, H, T, D, device="cuda", dtype=torch.bfloat16)
    caches = [torch.randn(1, KVH, S, D, device="cuda", dtype=torch.bfloat16) for _ in range(2 * LAYERS)]
    mask = (torch.arange(S, device="cuda") < valid)[None, None, None, :].contiguous()
    lengths = torch.tensor([valid], dtype=torch.int32, device="cuda")
    ref = q
    for i in range(LAYERS):
        ref = F.scaled_dot_product_attention(ref, caches[2 * i], caches[2 * i + 1], attn_mask=mask, enable_gqa=True)
    for label, use_mask, use_lengths, route in (
        ("bool mask                  ", True, False, ""),
        ("lengths                    ", False, True, ""),
        ("lengths, -mha_fusion:v1=off", False, True, "-mha_fusion:v1=off"),
        ("bool mask, -mha_fusion:v1=off", True, False, "-mha_fusion:v1=off"),
    ):
        engine, ctx = build(T, S, use_mask, use_lengths, route)
        out = torch.empty(1, H, T, D, device="cuda", dtype=torch.bfloat16)
        tensors = {"q": q, **{f"c{i}": c for i, c in enumerate(caches)}, engine.get_tensor_name(engine.num_io_tensors - 1): out}
        if use_mask:
            tensors["mask"] = mask
        if use_lengths:
            tensors["lengths"] = lengths
        for name, t in tensors.items():
            ctx.set_tensor_address(name, t.data_ptr())
        stream = torch.cuda.current_stream().cuda_stream
        run = lambda: ctx.execute_async_v3(stream)
        us = us_per_layer(run)
        err = ((out.float() - ref.float()).norm() / ref.float().norm()).item()
        print(f"T={T:2d} S={S}  {label}  {us:6.1f} us/layer  error {err:.4f}  {top_kernel(run)}", flush=True)
        del ctx, engine

Output:

$ python repro.py
11.4.0.106 NVIDIA PG509-210
T= 1 S=4096  bool mask                     167.5 us/layer  error 0.0036  _gemv_mha_v1
T= 1 S=4096  lengths                       295.5 us/layer  error 0.0036  _gemv_mha_v1
T= 1 S=4096  lengths, -mha_fusion:v1=off    35.2 us/layer  error 0.0031  _gemv_mha_v2
T= 1 S=4096  bool mask, -mha_fusion:v1=off    37.4 us/layer  error 0.3489  _gemv_mha_v2
T= 1 S=8192  bool mask                     324.3 us/layer  error 0.0034  _gemv_mha_v1
T= 1 S=8192  lengths                       577.1 us/layer  error 0.0034  _gemv_mha_v1
T= 1 S=8192  lengths, -mha_fusion:v1=off    42.5 us/layer  error 0.0030  _gemv_mha_v2
T= 1 S=8192  bool mask, -mha_fusion:v1=off    45.1 us/layer  error 0.3439  _gemv_mha_v2
T=64 S=4096  bool mask                     158.9 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  lengths                       139.8 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  lengths, -mha_fusion:v1=off   146.0 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  bool mask, -mha_fusion:v1=off   159.3 us/layer  error 0.0010  _gemm_mha_v2

Environment

TensorRT Version: 11.4.0.106 (also 11.5.0.63)

NVIDIA GPU: A100 80GB SXM (sm_80)

NVIDIA Driver Version: 580.126.09

CUDA Version: 13.2 (PyTorch wheel)

Operating System: Linux x86_64

Python Version: 3.12

PyTorch Version: 2.15.0+cu132 (only to make inputs and the reference)

Ngôn ngữ chính
C++
Star
13.5k
Fork
2.4k
Merge trung bình
6 ngày 21 giờ
Pull request đã merge (30 ngày)
3

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/TensorRT

Tất cả issue của NVIDIA/TensorRT

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.