Performance: fused MHA reaches 77-125 TFLOPS for BERT-large attention on RTX PRO 6000 Blackwell, 2-4x below FlashAttention-2 on the same GPU

Aperta
#4,881 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, python

Direzione di ricerca

The slow kernel is _gemm_mha_v2 in the trtexec layer profile, and the reproduction is the trtexec command sequence in the issue body with the profile JSON files. Start by reproducing the FP8 engine per-layer timings at 256 and 640 tokens. Finding why the kernel choice depends on the surrounding graph would mean working inside TensorRT's kernel selection, which is closed source. Done would mean the fused MHA reaching FlashAttention-2 throughput on the same shapes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Module:Performance

Description

In BERT-style encoders on an RTX PRO 6000 Blackwell Server Edition (SM120), TensorRT's fused attention kernel (_gemm_mha_v2 in the layer profile) runs at 77-125 TFLOPS. FlashAttention-2 on the same GPU and shapes (through vLLM's encoder attention) runs at 204-332 TFLOPS. The gap grows with the sequence length, and attention ends up as the largest part of the engine: 66.5 ms of the FP8 engine's 160.8 ms at 128 sequences x 640 tokens.

Model: BAAI/bge-reranker-v2-m3 (XLM-R large: 24 layers, 16 heads of 64), 128 sequences without padding (attention mask all ones), FP8 and NVFP4 engines (ModelOpt's default configs quantize only the Linear layers, so the attention inputs are FP16 in both). Time of the fused MHA kernel per layer from trtexec --dumpProfile --separateProfileRun, and TFLOPS counting QK^T and PV only (4 x 128 x 16 x L^2 x 64 FLOP per layer):

tokens per sequence FP8 engine NVFP4 engine FlashAttention-2 (flash_fwd_kernel in vLLM, same GPU)
256 0.297 ms (116 TFLOPS) 0.502 ms (68)
320 0.263 ms (204)
384 0.730 ms (106) 1.036 ms (75)
448 0.843 ms (125) 1.366 ms (77)
512 1.386 ms (99) 1.744 ms (79)
576 2.037 ms (85) 2.172 ms (80)
640 2.774 ms (77) 2.632 ms (82) 0.645 ms (332)
768 3.873 ms (80) 3.714 ms (83)
  • The attention inputs are the same in both engines, yet the NVFP4 engine's MHA is slower than the FP8 engine's up to 512 tokens, so the kernel choice seems to depend on the surrounding graph.
  • Builder optimization level 3 (default), TensorRT 11.3.0.99 from the Triton 26.09 image. TensorRT 11.4.0.106 builds the same kernel mix with the same engine times at 320 and 640 tokens.

Environment

TensorRT Version: 11.3.0.99 (also 11.4.0.106)

NVIDIA GPU: RTX PRO 6000 Blackwell Server Edition (SM120)

NVIDIA Driver Version: 595.91.07

CUDA Version: 13.4

CUDNN Version: not used

Operating System: Ubuntu 24.04 (GCP g4-standard-48)

Python Version (if applicable): 3.12

Baremetal or Container (if so, version): nvcr.io/nvidia/tritonserver:26.09-py3

Relevant Files

Model link: BAAI/bge-reranker-v2-m3, quantized with ModelOpt (FP8_DEFAULT_CFG, or NVFP4_DEFAULT_CFG with Q/K/V exported as one MatMul, see #4879) and exported with get_onnx_bytes_and_metadata(..., weights_dtype="fp16"). The ONNX files (about 700 MB each) can be shared on request.

Steps To Reproduce

trtexec --onnx=model.onnx --saveEngine=model.plan --minShapes=input_ids:1x128,attention_mask:1x128 \
  --optShapes=input_ids:32x640,attention_mask:32x640 --maxShapes=input_ids:128x768,attention_mask:128x768 \
  --profilingVerbosity=detailed --skipInference
# token ids in [1000, 200000), attention mask all ones, int64
for L in 256 384 448 512 576 640 768; do
  trtexec --loadEngine=model.plan --shapes=input_ids:128x$L,attention_mask:128x$L \
    --loadInputs=input_ids:ids-$L.bin,attention_mask:mask-$L.bin --warmUp=1000 --duration=5 --noDataTransfers \
    --useSpinWait --separateProfileRun --profilingVerbosity=detailed --exportProfile=profile-$L.json
done

The _gemm_mha_v2 rows of each profile give the per-layer time.

Have you tried the latest release?: yes, 11.4.0.106.

Lingua principale
C++
Stelle
13.5k
Fork
2.4k
Merge medio
6g 21h
PR unite (30g)
3

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT

Tutte le issue di NVIDIA/TensorRT

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.