Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Common] A TE built against cuBLAS ≥ 13.3 fails to import once an older cuBLASLt is loaded, so the runtime grouped-GEMM version check never runs

Đang mở
  • Có gì không ổn?
#3,751 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
c, python, pytorch
Lĩnh vực
backend, machine-learning

Hướng nghiên cứu

Start with common/gemm/cublaslt_grouped_gemm.cu, common/CMakeLists.txt, and the library loading path in _load_core_library(); compare how their symbol binding interacts with the runtime guard in pytorch/module/grouped_linear.py. Reproduce the import order with torch loaded first and CUDA_VISIBLE_DEVICES="". Done when import no longer fails with an undefined grouped GEMM symbol and the existing unsupported-version error path remains reachable.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Describe the bug

If TE is built against cuBLAS ≥ 13.3 and an older cuBLASLt is already loaded, import transformer_engine fails with an undefined grouped GEMM symbol. Because of that failure, the runtime cuBLASLt version check in GroupedLinear never gets a chance to run.

To reproduce:

  1. Build TE from source against a CUDA 13.2 or 13.3 toolkit, whose cuBLAS is 13.4.0.1 or 13.6.0.2.
  2. Install it into an environment whose PyTorch wheel brings an older cuBLASLt. For example, torch 2.14.1+cu130 depends on nvidia-cublas 13.1.1.3.
  3. Import torch first, then TE:
import torch
import transformer_engine.pytorch

OSError: .../transformer_engine/libtransformer_engine.so: undefined symbol:
cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13

Importing transformer_engine.pytorch before torch works, because libcublasLt.so.13 then resolves to the toolkit copy. No GPU is needed: CUDA_VISIBLE_DEVICES="" reproduces it. I reproduced it on main fac8d4a9 on aarch64 (GH200 host, CUDA 13.2) and on x86_64 (L40S host, CUDA 13.3). Several users report the same error in the comments of #2504.

Why it happens

  • libtransformer_engine.so links libcublasLt.so.13 by soname. The grouped GEMM path (common/gemm/cublaslt_grouped_gemm.cu) calls symbols that exist only in cuBLAS ≥ 13.3. If PyTorch has already loaded its cuBLASLt 13.1 under the same soname, TE's DT_NEEDED entry reuses that library.
  • The references are ordinary PLT (JUMP_SLOT) relocations, so lazy binding would defer them until the first call. _load_core_library() asks for lazy binding with os.RTLD_LAZY, which was added in #3127. Before #3127 the loader passed only RTLD_GLOBAL, and ctypes then binds eagerly. That matches the dates of the #2504 reports (April–May 2026).
  • Since #3383, the library is also linked with -z now (LINKER:-z,now in common/CMakeLists.txt; readelf -d shows FLAGS BIND_NOW and FLAGS_1 NOW). This forces eager binding whatever flag the loader passes, so os.RTLD_LAZY no longer has any effect.
  • As a result, the runtime guard in pytorch/module/grouped_linear.py (get_cublasLt_version() < 130300, and < 130400 below sm_100) can never run.

A toy library with one unresolved PLT call, loaded through ctypes.CDLL with glibc, shows the three cases:

Library RTLD_GLOBAL RTLD_GLOBAL | os.RTLD_LAZY
default link undefined symbol loads
-Wl,-z,now undefined symbol undefined symbol

Loading the toolkit's libcublasLt.so.13 with ctypes.CDLL(..., RTLD_GLOBAL) after import torch makes the import succeed. However, that leaves two cuBLASLt copies in the process, so I don't think it is a fix.

Possible directions (I'm happy to help with whichever you prefer)

  1. Resolve the grouped GEMM cuBLASLt entry points with dlsym at first use, the way #3434 did for NCCL EP. This works with any link flags. If the symbols are missing, use the existing "requires cuBLAS 13.3+" error path.
  2. Before loading libtransformer_engine.so, compare cublasLtGetVersion() of the cuBLASLt already in the process with the version TE was built against, and raise a clear error that names both versions and the import-order workaround. The build-time version would have to be recorded outside the library (for example, in the package metadata), because get_cublasLt_build_version() is only reachable after the load.

Environment

  • TE main fac8d4a9, built with NVTE_FRAMEWORK=pytorch NVTE_WITH_NCCL_EP=0
  • torch 2.14.1+cu130 (nvidia-cublas 13.1.1.3)
  • GH200 host: CUDA 13.2 toolkit (cuBLAS 13.4.0.1), aarch64
  • L40S host: CUDA 13.3 toolkit (cuBLAS 13.6.0.2), x86_64
Ngôn ngữ chính
Python
Star
3.6k
Fork
883
Merge trung bình
4 ngày 20 giờ
Pull request đã merge (30 ngày)
71

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/TransformerEngine

Tất cả issue của NVIDIA/TransformerEngine

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.