Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[BUG] lmp entry point does not preload nvJitLink, so the wheels' cuSPARSE and cuSOLVER can bind an older toolkit libnvJitLink

オープン 初心者向け
  • 何か問題がありますか？
#6,086 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
73/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python, pytorch
領域
backend, devops

調査の方向性

deepmd/lmp.pyから始めて、get_library_pathを通じてLD_PRELOADリストがどのように組み立てられるかを調べてください。報告では、不足しているライブラリnvidia.nvjitlink.libと必要なロード順序が示されています。提供されたLD_BIND_NOW=1コマンドと、LD_LIBRARY_PATH上のCUDA 12.8 toolkitを使って再現してください。wheelのnvJitLinkをcuSOLVERおよびcuSPARSEより先にプリロードすることで、報告されたシンボル検索エラーが回避できれば完了です。

索引モデルが issue の本文から書いたものです。

説明

Bug summary

deepmd/lmp.py, which sets up the environment of the lmp entry point of deepmd-kit[lmp], puts the CUDA libraries of the NVIDIA wheels into LD_PRELOAD: cudart, cublasLt, cublas, cufft, curand, cusolver, cusparse and cudnn. libnvJitLink is missing from that list, although three of the preloaded wheel libraries have it as a DT_NEEDED dependency: libcusparse.so.12, libcusolver.so.11 and libcusolverMg.so.11.

The dynamic loader resolves that dependency through LD_LIBRARY_PATH before it tries the wheel library's own RUNPATH ($ORIGIN/../../nvjitlink/lib). Clusters often put a CUDA toolkit on LD_LIBRARY_PATH, for example through a cuda/12.x module. If that toolkit is older than the CUDA of the torch wheel, the preloaded cuSPARSE binds the toolkit's older libnvJitLink.so.12. That library lacks the versioned symbols of the newer release, so the process fails with a symbol lookup error.

DeePMD-kit Version

master (668b60cc2); deepmd/lmp.py is unchanged since #5707

Backend and its version

PyTorch 2.13.0+cu129 (pip, with the NVIDIA CUDA wheels); CUDA toolkit 12.8 on LD_LIBRARY_PATH; driver 535.161.08

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

We hit it while porting to an H20 node, in a LAMMPS built from source against the same pip torch (not the deepmd-kit[lmp] wheel), when DeePMD-kit loaded the PyTorch exportable backend for a .pt2 model:

Unable to load PyTorch Exportable backend plugin (libdeepmd_backend_ptexpt.so) ... nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12

We did not run the deepmd-kit[lmp] wheel itself; how it is affected follows from the preload list and the reproduction below. A Python process is not affected, because torch loads the libraries of its own wheels first.

Steps to Reproduce

SITE is the site-packages of an environment with torch==2.13.0+cu129 and its NVIDIA wheels; /usr/local/cuda is a CUDA 12.8 toolkit. Test 1 preloads cuSPARSE the way deepmd/lmp.py does, with the older toolkit on LD_LIBRARY_PATH:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# /bin/true: symbol lookup error: .../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12
# exit status 127

Test 2 preloads the wheel's nvJitLink first:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/nvjitlink/lib/libnvJitLink.so.12:$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# exit status 0

LD_BIND_NOW=1 makes the error appear at startup. Without it, the error appears when the symbols are first resolved; in our run, that was when the backend plugin was loaded.

The wheel libraries that need nvJitLink can be listed with readelf -d:

for f in $SITE/nvidia/*/lib/lib*.so.*; do readelf -d "$f" | grep -q 'NEEDED.*libnvJitLink' && echo "$f"; done
# .../nvidia/cusolver/lib/libcusolver.so.11
# .../nvidia/cusolver/lib/libcusolverMg.so.11
# .../nvidia/cusparse/lib/libcusparse.so.12
Further Information, Files, and Links

Suggested fix: preload the wheel's nvJitLink ahead of cuSOLVER and cuSPARSE. LD_PRELOAD loads its entries in order, and a library that is already loaded satisfies a later DT_NEEDED with the same soname:

*get_library_path("nvidia.cuda_runtime.lib", "libcudart.so*"),
*get_library_path("nvidia.nvjitlink.lib", "libnvJitLink.so*"),
*get_library_path("nvidia.cublas.lib", "libcublasLt.so*"),
...

A related observation, which we have not traced further: the CUDA 13 builds of PyTorch (seen with torch==2.13.0+cu130) install their CUDA libraries into a single nvidia/cu13/lib directory, with no nvidia.cuda_runtime, nvidia.cusparse, ... packages. With such a torch, get_library_path matches none of the entries above, and nothing is preloaded.

主要言語
Python
スター
2.1k
フォーク
659
平均マージ
4日 13時間
マージ済み PR（30日）
9

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

deepmodeling/deepmd-kit のほかの issue

deepmodeling/deepmd-kit の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。