bf16 decode attention picks `_gemv_mha_v1`, 5x to 8x slower than `_gemv_mha_v2`, and `_gemv_mha_v2` is wrong with a bool mask

Ouverte
#4,947 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 6 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
28/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Domaine
backend, performance

Piste de recherche

The issue provides a TensorRT Python API repro and names _gemv_mha_v1, _gemv_mha_v2, and _gemm_mha_v2, but no implementation files or tests. Start by running the repro on the stated TensorRT and A100 setup to confirm the timing and mask errors. Done means selecting a fast split path for long caches, handling bool masks correctly, and adding a split path for a few queries; this requires kernel and dispatch investigation.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Module:Performance

What is wrong

In decoding, attention has 1 query over a long key/value (KV) cache. For a bfloat16 IAttention layer on A100, the builder picks _gemv_mha_v1. It launches one thread block per query head (16 blocks on a GPU with 108 multiprocessors). At 4096 slots it takes 168 us per layer for 16.8 MB of keys and values, about 100 GB/s.

TensorRT also has _gemv_mha_v2, which splits the cache across many blocks (256 at 4096 slots, 512 at 8192). When I force it with the build route -mha_fusion:v1=off, it is 5x to 8x faster. But it gives correct results only when the valid length comes from key_value_lengths. With a bool mask the result is wrong (error 0.35), which fits the mask being ignored. In float16 TensorRT picks this kernel by default, and there the mask is ignored (pytorch/TensorRT#5011, measured on H100 above 512 keys). Passing key_value_lengths without the route makes the default kernel 1.8x slower.

1 query, 16 heads, 8 KV heads, head size 128 4096 slots 8192 slots
bool mask (default) 168 us, correct 324 us, correct
key_value_lengths (default) 296 us, correct 577 us, correct
key_value_lengths, -mha_fusion:v1=off 35 us, correct 43 us, correct
bool mask, -mha_fusion:v1=off 37 us, wrong 45 us, wrong

For reference, FlashAttention's split-KV kernel (flash_attn_with_kvcache, flash-attn 2.8.3, same GPU and valid length) takes 23 us and 35 us on the same shapes. With 64 queries, _gemm_mha_v2 also launches 16 blocks, and the layer takes 140 to 159 us (including 2 copies of K and V), against 28 us. No build route I tried gave it a split path.

At 4096 slots TensorRT 11.5 picks the same default kernels with the same times. With the route and a bool mask it falls back to a slower kernel (165 us) and is correct.

I found this running Alpamayo 2 Edge through Torch-TensorRT (pytorch/executorch#25253). There, my measurements put the whole gap to PyTorch with FlashAttention on A100 in attention for a few queries over the cache, in decoding and in the action head.

Expected

  • For 1 query over a long cache, pick the split kernel by default.
  • _gemv_mha_v2 either applies a bool mask or is never picked with one.
  • A split path for a few queries (tens) over a long cache.

Repro

TensorRT Python API only. 28 attention layers in a chain, so they cannot overlap.

repro.py
"""bf16 attention over a long KV cache with the TensorRT Python API (no other framework).

28 attention layers in a chain (each output is the next query), 16 query heads, 8 KV heads, head size 128.
The cache has S slots and the first 7/8 of them are valid. Each case gives the valid part either as a bool
mask [1, 1, 1, S] or as key_value_lengths. Times are per layer; error is relative to PyTorch on the same inputs.
"""
import math

import numpy as np
import tensorrt as trt
import torch
import torch.nn.functional as F

H, KVH, D, LAYERS = 16, 8, 128, 28
logger = trt.Logger(trt.Logger.WARNING)


def build(T, S, use_mask, use_lengths, route):
    builder = trt.Builder(logger)
    net = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
    x = net.add_input("q", trt.bfloat16, (1, H, T, D))
    caches = [net.add_input(f"c{i}", trt.bfloat16, (1, KVH, S, D)) for i in range(2 * LAYERS)]
    mask = net.add_input("mask", trt.bool, (1, 1, 1, S)) if use_mask else None
    lengths = net.add_input("lengths", trt.int32, (1,)) if use_lengths else None
    scale = net.add_constant((1, 1, 1, 1), trt.Weights(np.array([1 / math.sqrt(D)], dtype=np.float32))).get_output(0)
    scale = net.add_cast(scale, trt.bfloat16).get_output(0)
    for i in range(LAYERS):
        q = net.add_elementwise(x, scale, trt.ElementWiseOperation.PROD).get_output(0)
        att = net.add_attention_v2(q, caches[2 * i], caches[2 * i + 1], trt.AttentionNormalizationOp.SOFTMAX, trt.CausalMaskKind.NONE)
        if mask is not None:
            att.mask = mask
        if lengths is not None:
            att.key_value_lengths = lengths
        x = att.get_output(0)
    net.mark_output(x)
    config = builder.create_builder_config()
    if route:
        config.build_route = route
    engine = trt.Runtime(logger).deserialize_cuda_engine(builder.build_serialized_network(net, config))
    return engine, engine.create_execution_context()


def top_kernel(fn):
    with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA]) as prof:
        fn()
        torch.cuda.synchronize()
    k = max((e for e in prof.key_averages() if e.device_time_total > 0), key=lambda e: e.device_time_total)
    return k.key.split("_0x")[0]


def us_per_layer(fn, iters=200):
    for _ in range(10):
        fn()
    torch.cuda.synchronize()
    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    s.record()
    for _ in range(iters):
        fn()
    e.record()
    torch.cuda.synchronize()
    return s.elapsed_time(e) / iters / LAYERS * 1000


print(trt.__version__, torch.cuda.get_device_name())
torch.cuda.set_stream(torch.cuda.Stream())  # TensorRT warns about the default stream
torch.manual_seed(0)
for T, S in ((1, 4096), (1, 8192), (64, 4096)):
    valid = S - S // 8
    q = torch.randn(1, H, T, D, device="cuda", dtype=torch.bfloat16)
    caches = [torch.randn(1, KVH, S, D, device="cuda", dtype=torch.bfloat16) for _ in range(2 * LAYERS)]
    mask = (torch.arange(S, device="cuda") < valid)[None, None, None, :].contiguous()
    lengths = torch.tensor([valid], dtype=torch.int32, device="cuda")
    ref = q
    for i in range(LAYERS):
        ref = F.scaled_dot_product_attention(ref, caches[2 * i], caches[2 * i + 1], attn_mask=mask, enable_gqa=True)
    for label, use_mask, use_lengths, route in (
        ("bool mask                  ", True, False, ""),
        ("lengths                    ", False, True, ""),
        ("lengths, -mha_fusion:v1=off", False, True, "-mha_fusion:v1=off"),
        ("bool mask, -mha_fusion:v1=off", True, False, "-mha_fusion:v1=off"),
    ):
        engine, ctx = build(T, S, use_mask, use_lengths, route)
        out = torch.empty(1, H, T, D, device="cuda", dtype=torch.bfloat16)
        tensors = {"q": q, **{f"c{i}": c for i, c in enumerate(caches)}, engine.get_tensor_name(engine.num_io_tensors - 1): out}
        if use_mask:
            tensors["mask"] = mask
        if use_lengths:
            tensors["lengths"] = lengths
        for name, t in tensors.items():
            ctx.set_tensor_address(name, t.data_ptr())
        stream = torch.cuda.current_stream().cuda_stream
        run = lambda: ctx.execute_async_v3(stream)
        us = us_per_layer(run)
        err = ((out.float() - ref.float()).norm() / ref.float().norm()).item()
        print(f"T={T:2d} S={S}  {label}  {us:6.1f} us/layer  error {err:.4f}  {top_kernel(run)}", flush=True)
        del ctx, engine

Output:

$ python repro.py
11.4.0.106 NVIDIA PG509-210
T= 1 S=4096  bool mask                     167.5 us/layer  error 0.0036  _gemv_mha_v1
T= 1 S=4096  lengths                       295.5 us/layer  error 0.0036  _gemv_mha_v1
T= 1 S=4096  lengths, -mha_fusion:v1=off    35.2 us/layer  error 0.0031  _gemv_mha_v2
T= 1 S=4096  bool mask, -mha_fusion:v1=off    37.4 us/layer  error 0.3489  _gemv_mha_v2
T= 1 S=8192  bool mask                     324.3 us/layer  error 0.0034  _gemv_mha_v1
T= 1 S=8192  lengths                       577.1 us/layer  error 0.0034  _gemv_mha_v1
T= 1 S=8192  lengths, -mha_fusion:v1=off    42.5 us/layer  error 0.0030  _gemv_mha_v2
T= 1 S=8192  bool mask, -mha_fusion:v1=off    45.1 us/layer  error 0.3439  _gemv_mha_v2
T=64 S=4096  bool mask                     158.9 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  lengths                       139.8 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  lengths, -mha_fusion:v1=off   146.0 us/layer  error 0.0010  _gemm_mha_v2
T=64 S=4096  bool mask, -mha_fusion:v1=off   159.3 us/layer  error 0.0010  _gemm_mha_v2

Environment

TensorRT Version: 11.4.0.106 (also 11.5.0.63)

NVIDIA GPU: A100 80GB SXM (sm_80)

NVIDIA Driver Version: 580.126.09

CUDA Version: 13.2 (PyTorch wheel)

Operating System: Linux x86_64

Python Version: 3.12

PyTorch Version: 2.15.0+cu132 (only to make inputs and the reference)

Langage dominant
C++
Étoiles
13.5k
Forks
2.4k
Merge moyen
6 j 21 h
PR mergées (30 j)
3

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/TensorRT

Toutes les issues de NVIDIA/TensorRT

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.