bf16 decode attention picks `_gemv_mha_v1`, 5x to 8x slower than `_gemv_mha_v2`, and `_gemv_mha_v2` is wrong with a bool mask

未关闭
#4,947 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 6 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
28/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃

调研方向

The issue provides a TensorRT Python API repro and names _gemv_mha_v1, _gemv_mha_v2, and _gemm_mha_v2, but no implementation files or tests. Start by running the repro on the stated TensorRT and A100 setup to confirm the timing and mask errors. Done means selecting a fast split path for long caches, handling bool masks correctly, and adding a split path for a few queries; this requires kernel and dispatch investigation.

由索引模型根据 Issue 内容生成。

描述

Module:Performance

What is wrong

In decoding, attention has 1 query over a long key/value (KV) cache. For a bfloat16 IAttention layer on A100, the builder picks _gemv_mha_v1. It launches one thread block per query head (16 blocks on a GPU with 108 multiprocessors). At 4096 slots it takes 168 us per layer for 16.8 MB of keys and values, about 100 GB/s.

TensorRT also has _gemv_mha_v2, which splits the cache across many blocks (256 at 4096 slots, 512 at 8192). When I force it with the build route -mha_fusion:v1=off, it is 5x to 8x faster. But it gives correct results only when the valid length comes from key_value_lengths. With a bool mask the result is wrong (error 0.35), which fits the mask being ignored. In float16 TensorRT picks this kernel by default, and there the mask is ignored (pytorch/TensorRT#5011, measured on H100 above 512 keys). Passing key_value_lengths without the route makes the default kernel 1.8x slower.

1 query, 16 heads, 8 KV heads, head size 128 4096 slots 8192 slots
bool mask (default) 168 us, correct 324 us, correct
key_value_lengths (default) 296 us, correct 577 us, correct
key_value_lengths, -mha_fusion:v1=off 35 us, correct 43 us, correct
bool mask, -mha_fusion:v1=off 37 us, wrong 45 us, wrong

For reference, FlashAttention's split-KV kernel (flash_attn_with_kvcache, flash-attn 2.8.3, same GPU and valid length) takes 23 us and 35 us on the same shapes. With 64 queries, _gemm_mha_v2 also launches 16 blocks, and the layer takes 140 to 159 us (including 2 copies of K and V), against 28 us. No build route I tried gave it a split path.

At 4096 slots TensorRT 11.5 picks the same default kernels with the same times. With the route and a bool mask it falls back to a slower kernel (165 us) and is correct.

I found this running Alpamayo 2 Edge through Torch-TensorRT (pytorch/executorch#25253). There, my measurements put the whole gap to PyTorch with FlashAttention on A100 in attention for a few queries over the cache, in decoding and in the action head.

Expected

  • For 1 query over a long cache, pick the split kernel by default.
  • _gemv_mha_v2 either applies a bool mask or is never picked with one.
  • A split path for a few queries (tens) over a long cache.

Repro

TensorRT Python API only. 28 attention layers in a chain, so they cannot overlap.

repro.py
"""bf16 attention over a long KV cache with the TensorRT Python API (no other framework).

28 attention layers in a chain (each output is the next query), 16 query heads, 8 KV heads, head size 128.
The cache has S slots and the first 7/8 of them are valid. Each case gives the valid part either as a bool
mask [1, 1, 1, S] or as key_value_lengths. Times are per layer; error is relative to PyTorch on the same inputs.
"""
import math

import numpy as np
import tensorrt as trt
import torch
import torch.nn.functional as F

H, KVH, D, LAYERS = 16, 8, 128, 28
logger = trt.Logger(trt.Logger.WARNING)


def build(T, S, use_mask, use_lengths, route):
    builder = trt.Builder(logger)
    net = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
    x = net.add_input("q", trt.bfloat16, (1, H, T, D))
    caches = [net.add_input(f"c{i}", trt.bfloat16, (1, KVH, S, D)) for i in range(2 * LAYERS)]
    mask = net.add_input("mask", trt.bool, (1, 1, 1, S)) if use_mask else None
    lengths = net.add_input("lengths", trt.int32, (1,)) if use_lengths else None
    scale = net.add_constant((1, 1, 1, 1), trt.Weights(np.array([1 / math.sqrt(D)], dtype=np.float32))).get_output(0)
    scale = net.add_cast(scale, trt.bfloat16).get_output(0)
    for i in range(LAYERS):
        q = net.add_elementwise(x, scale, trt.ElementWiseOperation.PROD).get_output(0)
        att = net.add_attention_v2(q, caches[2 * i], caches[2 * i + 1], trt.AttentionNormalizationOp.SOFTMAX, trt.CausalMaskKind.NONE)
        if mask is not None:
            att.mask = mask
        if lengths is not None:
            att.key_value_lengths = lengths
        x = att.get_output(0)
    net.mark_output(x)
    config = builder.create_builder_config()
    if route:
        config.build_route = route
    engine = trt.Runtime(logger).deserialize_cuda_engine(builder.build_serialized_network(net, config))
    return engine, engine.create_execution_context()


def top_kernel(fn):
    with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA]) as prof:
        fn()
        torch.cuda.synchronize()
    k = max((e for e in prof.key_averages() if e.device_time_total > 0), key=lambda e: e.device_time_total)
    return k.key.split("_0x")[0]


def us_per_layer(fn, iters=200):
    for _ in range(10):
        fn()
    torch.cuda.synchronize()
    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    s.record()
    for _ in range(iters):
        fn()
    e.record()
    torch.cuda.synchronize()
    return s.elapsed_time(e) / iters / LAYERS * 1000


print(trt.__version__, torch.cuda.get_device_name())
torch.cuda.set_stream(torch.cuda.Stream())  # TensorRT warns about the default stream
torch.manual_seed(0)
for T, S in ((1, 4096), (1, 8192), (64, 4096)):
    valid = S - S // 8
    q = torch.randn(1, H, T, D, device="cuda", dtype=torch.bfloat16)
    caches = [torch.randn(1, KVH, S, D, device="cuda", dtype=torch.bfloat16) for _ in range(2 * LAYERS)]
    mask = (torch.arange(S, device="cuda") < valid)[None, None, None, :].contiguous()
    lengths = torch.tensor([valid], dtype=torch.int32, device="cuda")
    ref = q
    for i in range(LAYERS):
        ref = F.scaled_dot_product_attention(ref, caches[2 * i], caches[2 * i + 1], attn_mask=mask, enable_gqa=True)
    for label, use_mask, use_lengths, route in (
        ("bool mask                  ", True, False, ""),
        ("lengths                    ", False, True, ""),
        ("lengths, -mha_fusion:v1=off", False, True, "-mha_fusion:v1=off"),
        ("bool mask, -mha_fusion:v1=off", True, False, "-mha_fusion:v1=off"),
    ):
        engine, ctx = build(T, S, use_mask, use_lengths, route)
        out = torch.empty(1, H, T, D, device="cuda", dtype=torch.bfloat16)
        tensors = {"q": q, **{f"c{i}": c for i, c in enumerate(caches)}, engine.get_tensor_name(engine.num_io_tensors - 1): out}
        if use_mask:
            tensors["mask"] = mask
        if use_lengths:
            tensors["lengths"] = lengths
        for name, t in tensors.items():
            ctx.set_tensor_address(name, t.data_ptr())
        stream = torch.cuda.current_stream().cuda_stream
        run = lambda: ctx.execute_async_v3(stream)
        us = us_per_layer(run)
        err = ((out.float() - ref.float()).norm() / ref.float().norm()).item()
        print(f"T={T:2d} S={S}  {label}  {us:6.1f} us/layer  error {err:.4f}  {top_kernel(run)}", flush=True)
        del ctx, engine

Output:

$ python repro.py
11.4.0.106 NVIDIA PG509-210
T= 1 S=4096  bool mask                     167.5 us/layer  error 0.0036  _gemv_mha_v1
T= 1 S=4096  lengths                       295.5 us/layer  error 0.0036  _gemv_mha_v1
T= 1 S=4096  lengths, -mha_fusion:v1=off    35.2 us/layer  error 0.0031  _gemv_mha_v2
T= 1 S=4096  bool mask, -mha_fusion:v1=off    37.4 us/layer  error 0.3489  _gemv_mha_v2
T= 1 S=8192  bool mask                     324.3 us/layer  error 0.0034  _gemv_mha_v1
T= 1 S=8192  lengths                       577.1 us/layer  error 0.0034  _gemv_mha_v1
T= 1 S=8192  lengths, -mha_fusion:v1=off    42.5 us/layer  error 0.0030  _gemv_mha_v2
T= 1 S=8192  bool mask, -mha_fusion:v1=off    45.1 us/layer  error 0.3439  _gemv_mha_v2
T=64 S=4096  bool mask                     158.9 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  lengths                       139.8 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  lengths, -mha_fusion:v1=off   146.0 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  bool mask, -mha_fusion:v1=off   159.3 us/layer  error 0.0010  _gemm_mha_v2

Environment

TensorRT Version: 11.4.0.106 (also 11.5.0.63)

NVIDIA GPU: A100 80GB SXM (sm_80)

NVIDIA Driver Version: 580.126.09

CUDA Version: 13.2 (PyTorch wheel)

Operating System: Linux x86_64

Python Version: 3.12

PyTorch Version: 2.15.0+cu132 (only to make inputs and the reference)

主要语言
C++
星标
13.5k
派生
2.4k
平均合并
6 天 21 小时
30 天内合并 PR
3

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/TensorRT 的其他 Issue

查看 NVIDIA/TensorRT 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。