Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Common] A TE built against cuBLAS ≥ 13.3 fails to import once an older cuBLASLt is loaded, so the runtime grouped-GEMM version check never runs

オープン
  • 何か問題がありますか？
#3,751 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
c, python, pytorch
領域
backend, machine-learning

調査の方向性

Start with common/gemm/cublaslt_grouped_gemm.cu, common/CMakeLists.txt, and the library loading path in _load_core_library(); compare how their symbol binding interacts with the runtime guard in pytorch/module/grouped_linear.py. Reproduce the import order with torch loaded first and CUDA_VISIBLE_DEVICES="". Done when import no longer fails with an undefined grouped GEMM symbol and the existing unsupported-version error path remains reachable.

索引モデルが issue の本文から書いたものです。

説明

Describe the bug

If TE is built against cuBLAS ≥ 13.3 and an older cuBLASLt is already loaded, import transformer_engine fails with an undefined grouped GEMM symbol. Because of that failure, the runtime cuBLASLt version check in GroupedLinear never gets a chance to run.

To reproduce:

  1. Build TE from source against a CUDA 13.2 or 13.3 toolkit, whose cuBLAS is 13.4.0.1 or 13.6.0.2.
  2. Install it into an environment whose PyTorch wheel brings an older cuBLASLt. For example, torch 2.14.1+cu130 depends on nvidia-cublas 13.1.1.3.
  3. Import torch first, then TE:
import torch
import transformer_engine.pytorch

OSError: .../transformer_engine/libtransformer_engine.so: undefined symbol:
cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13

Importing transformer_engine.pytorch before torch works, because libcublasLt.so.13 then resolves to the toolkit copy. No GPU is needed: CUDA_VISIBLE_DEVICES="" reproduces it. I reproduced it on main fac8d4a9 on aarch64 (GH200 host, CUDA 13.2) and on x86_64 (L40S host, CUDA 13.3). Several users report the same error in the comments of #2504.

Why it happens

  • libtransformer_engine.so links libcublasLt.so.13 by soname. The grouped GEMM path (common/gemm/cublaslt_grouped_gemm.cu) calls symbols that exist only in cuBLAS ≥ 13.3. If PyTorch has already loaded its cuBLASLt 13.1 under the same soname, TE's DT_NEEDED entry reuses that library.
  • The references are ordinary PLT (JUMP_SLOT) relocations, so lazy binding would defer them until the first call. _load_core_library() asks for lazy binding with os.RTLD_LAZY, which was added in #3127. Before #3127 the loader passed only RTLD_GLOBAL, and ctypes then binds eagerly. That matches the dates of the #2504 reports (April–May 2026).
  • Since #3383, the library is also linked with -z now (LINKER:-z,now in common/CMakeLists.txt; readelf -d shows FLAGS BIND_NOW and FLAGS_1 NOW). This forces eager binding whatever flag the loader passes, so os.RTLD_LAZY no longer has any effect.
  • As a result, the runtime guard in pytorch/module/grouped_linear.py (get_cublasLt_version() < 130300, and < 130400 below sm_100) can never run.

A toy library with one unresolved PLT call, loaded through ctypes.CDLL with glibc, shows the three cases:

Library RTLD_GLOBAL RTLD_GLOBAL | os.RTLD_LAZY
default link undefined symbol loads
-Wl,-z,now undefined symbol undefined symbol

Loading the toolkit's libcublasLt.so.13 with ctypes.CDLL(..., RTLD_GLOBAL) after import torch makes the import succeed. However, that leaves two cuBLASLt copies in the process, so I don't think it is a fix.

Possible directions (I'm happy to help with whichever you prefer)

  1. Resolve the grouped GEMM cuBLASLt entry points with dlsym at first use, the way #3434 did for NCCL EP. This works with any link flags. If the symbols are missing, use the existing "requires cuBLAS 13.3+" error path.
  2. Before loading libtransformer_engine.so, compare cublasLtGetVersion() of the cuBLASLt already in the process with the version TE was built against, and raise a clear error that names both versions and the import-order workaround. The build-time version would have to be recorded outside the library (for example, in the package metadata), because get_cublasLt_build_version() is only reachable after the load.

Environment

  • TE main fac8d4a9, built with NVTE_FRAMEWORK=pytorch NVTE_WITH_NCCL_EP=0
  • torch 2.14.1+cu130 (nvidia-cublas 13.1.1.3)
  • GH200 host: CUDA 13.2 toolkit (cuBLAS 13.4.0.1), aarch64
  • L40S host: CUDA 13.3 toolkit (cuBLAS 13.6.0.2), x86_64
主要言語
Python
スター
3.6k
フォーク
883
平均マージ
5日 37分
マージ済み PR（30日）
72

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/TransformerEngine のほかの issue

NVIDIA/TransformerEngine の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。