bfloat16 GEMM is noticeably slower than float16

オープン
#2,033 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
静か
技術スタック
cpp

調査の方向性

RTX 4090 ベンチマークを再現し、その後 src/cuda/primitives.cu の fp16 および bf16 GEMM 特殊化と、cublas_stub.cc の cuBLAS ロードコードを調査します。小さい方のトグルを、issue で説明されている cuBLASLt への移行と比較し、src/ops/conv1d_cudnn_gpu.cu における Allocator の使用も検討します。選択したアプローチによって、意図された数値動作を暗黙に変更することなく、報告されたワークロードに対する bf16 の実行時間が改善されれば完了です。

索引モデルが issue の本文から書いたものです。

説明

On an RTX 4090 (compute capability 8.9), running bert-base-uncased for masked-LM scoring with the prebuilt ctranslate2==4.7.1 wheel, bf16 inference is consistently slower than fp16 on the same inputs:

  • fp16, batch 64: ~34 ms per 30-sentence call
  • bf16, batch 64: ~42 ms per 30-sentence call (~24% slower)

The same workload run through PyTorch's BertForMaskedLM shows fp16 and bf16 within rounding error of each other (~32 ms each).

I've since investigated further with a Qwen3-0.6B embedding model. The cause turned out to be different from what I first wrote, so I've rewritten this issue.

The fp16/bf16 gap is expected on GeForce cards

CTranslate2's fp16 GEMMs accumulate in fp16 by default: CUBLAS_COMPUTE_16F, with CT2_CUDA_TRUE_FP16_GEMM on by default. On GeForce Ada cards, tensor-core math with fp32 accumulation runs at half the fp16-accumulate rate. On an RTX 4090 that's 165 vs 330 dense TFLOPS.

bf16 can only be accumulated in fp32. cuBLAS has no bf16 compute type, and the tensor cores have no bf16 accumulator. So bf16 can't get the speedup fp16 gets. The environment flag for bf16 accumulation that I suggested originally isn't possible.

PyTorch accumulates both in fp32, which is why its fp16 and bf16 tie. On data-center GPUs, where fp32 accumulation runs at full rate, the gap shouldn't appear.

For scale, at Qwen3-0.6B's linear-layer shapes (5096 rows), one layer's GEMMs take:

  • bf16: 1258 µs
  • fp16 with fp32 accumulation: 1271 µs
  • fp16 with fp16 accumulation: 877 µs

The real gap: CTranslate2 bf16 vs PyTorch bf16

The test: microsoft/harrier-oss-v1-0.6b (a Qwen3-0.6B embedding model) in bf16 on the RTX 4090. The workload is 591 texts of ~364 tokens in batches of 14, with medians of 5 runs.

  • PyTorch: sentence-transformers on torch 2.14.1+cu130 with SDPA.
  • CTranslate2: a 4.8.2 build of my fork that adds Qwen3-Embedding support by running the decoder as an encoder. The allocator, GEMM and decoder-layer code are identical to master.

Results:

  • Overall: CTranslate2 takes ~10% longer, 4.06 s vs 3.65 s.
  • Not the GEMMs: both run the same cuBLAS kernels (ampere_bf16_s1688gemm…) for the same total time, 1.50 s vs 1.45 s. Switching from cuBLAS to cuBLASLt changes these shapes by ~2%. PyTorch 2.14 itself uses cuBLAS here: torch.backends.cuda.preferred_blas_library() reports Cublas.
  • Idle GPU: CTranslate2 actually spends less time in GPU kernels, 2.96 s vs 3.19 s. But the GPU is idle about a third of the time: 68% busy vs 84% for PyTorch.

Part of that idle time comes from the default cuda_malloc_async allocator. Each batch makes ~630 cudaMallocAsync/cudaFreeAsync pairs. The pool's release threshold is never raised from its default of 0, so freed memory goes back to the driver at every stream synchronization.

CTranslate2 alone on the same workload:

Allocator 591 texts, batches of 14 One query (~25 tokens)
cuda_malloc_async (default) 4.15 s 16.0 ms
cuda_malloc_async, release threshold set to the maximum 3.96 s 16.2 ms
cub_caching, default configuration 17.97 s 10.6 ms
cub_caching, CT2_CUDA_CACHING_ALLOCATOR_CONFIG=2,8,30,4294967296 3.85 s 12.5 ms

The default cub_caching configuration caches nothing above 16 MB and holds at most 200 MB in total. That's why it's so slow with batches of long texts.

Against PyTorch, the tuned cub_caching takes CTranslate2 from 0.90x to 0.95x for batched texts, and from 1.11x to 1.45x for single queries. In one run with PyTorch loaded in the same process, that configuration crashed at exit (code 139). I couldn't reproduce it in a minimal script.

What might help

  • Allocator defaults. Both are small changes in src/cuda/allocator.cc:
    • when cuda_malloc_async is selected, raise cudaMemPoolAttrReleaseThreshold on its memory pool;
    • give cub_caching default bins and a cache size that cover transformer activations.
  • cuBLASLt. It's mainly worth it for fusing the bias (and GELU) into the GEMM epilogue in models with biases such as BERT, which PyTorch's addmm path does. For bias-free layers, cuBLASLt's algorithm selection alone gains ~2%.
  • The remaining gap. It looks like per-kernel overhead plus unfused attention. CTranslate2 runs a separate softmax and transposes where PyTorch uses one memory-efficient attention kernel.
  • A workaround meanwhile. On GeForce cards, running a bf16 model with compute_type="float16" gets the fp16-accumulate speedup, where the model tolerates fp16's range. For this model that was ~20% faster than bf16, and its embeddings were as close to PyTorch's bf16 output as CTranslate2's own bf16 run.

Happy to share the benchmark harness used to produce the numbers above if it'd help with reproduction.

主要言語
C++
スター
4.8k
フォーク
557
平均マージ
1日 6時間
マージ済み PR(30日)
24

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

OpenNMT/CTranslate2 のほかの issue

OpenNMT/CTranslate2 の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。