[BUG] lmp entry point does not preload nvJitLink, so the wheels' cuSPARSE and cuSOLVER can bind an older toolkit libnvJitLink

Open Beginner friendly
  • Is something wrong?
#6,086 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
73/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
backend, devops

Research direction

Start with deepmd/lmp.py and inspect how its LD_PRELOAD list is assembled through get_library_path; the report identifies the missing nvidia.nvjitlink.lib library and the required load order. Reproduce with the provided LD_BIND_NOW=1 commands and CUDA 12.8 toolkit on LD_LIBRARY_PATH. Done when preloading the wheel's nvJitLink ahead of cuSOLVER and cuSPARSE avoids the reported symbol lookup error.

Written by the indexing model from the issue text.

Description

Bug summary

deepmd/lmp.py, which sets up the environment of the lmp entry point of deepmd-kit[lmp], puts the CUDA libraries of the NVIDIA wheels into LD_PRELOAD: cudart, cublasLt, cublas, cufft, curand, cusolver, cusparse and cudnn. libnvJitLink is missing from that list, although three of the preloaded wheel libraries have it as a DT_NEEDED dependency: libcusparse.so.12, libcusolver.so.11 and libcusolverMg.so.11.

The dynamic loader resolves that dependency through LD_LIBRARY_PATH before it tries the wheel library's own RUNPATH ($ORIGIN/../../nvjitlink/lib). Clusters often put a CUDA toolkit on LD_LIBRARY_PATH, for example through a cuda/12.x module. If that toolkit is older than the CUDA of the torch wheel, the preloaded cuSPARSE binds the toolkit's older libnvJitLink.so.12. That library lacks the versioned symbols of the newer release, so the process fails with a symbol lookup error.

DeePMD-kit Version

master (668b60cc2); deepmd/lmp.py is unchanged since #5707

Backend and its version

PyTorch 2.13.0+cu129 (pip, with the NVIDIA CUDA wheels); CUDA toolkit 12.8 on LD_LIBRARY_PATH; driver 535.161.08

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

We hit it while porting to an H20 node, in a LAMMPS built from source against the same pip torch (not the deepmd-kit[lmp] wheel), when DeePMD-kit loaded the PyTorch exportable backend for a .pt2 model:

Unable to load PyTorch Exportable backend plugin (libdeepmd_backend_ptexpt.so) ... nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12

We did not run the deepmd-kit[lmp] wheel itself; how it is affected follows from the preload list and the reproduction below. A Python process is not affected, because torch loads the libraries of its own wheels first.

Steps to Reproduce

SITE is the site-packages of an environment with torch==2.13.0+cu129 and its NVIDIA wheels; /usr/local/cuda is a CUDA 12.8 toolkit. Test 1 preloads cuSPARSE the way deepmd/lmp.py does, with the older toolkit on LD_LIBRARY_PATH:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# /bin/true: symbol lookup error: .../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12
# exit status 127

Test 2 preloads the wheel's nvJitLink first:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/nvjitlink/lib/libnvJitLink.so.12:$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# exit status 0

LD_BIND_NOW=1 makes the error appear at startup. Without it, the error appears when the symbols are first resolved; in our run, that was when the backend plugin was loaded.

The wheel libraries that need nvJitLink can be listed with readelf -d:

for f in $SITE/nvidia/*/lib/lib*.so.*; do readelf -d "$f" | grep -q 'NEEDED.*libnvJitLink' && echo "$f"; done
# .../nvidia/cusolver/lib/libcusolver.so.11
# .../nvidia/cusolver/lib/libcusolverMg.so.11
# .../nvidia/cusparse/lib/libcusparse.so.12
Further Information, Files, and Links

Suggested fix: preload the wheel's nvJitLink ahead of cuSOLVER and cuSPARSE. LD_PRELOAD loads its entries in order, and a library that is already loaded satisfies a later DT_NEEDED with the same soname:

*get_library_path("nvidia.cuda_runtime.lib", "libcudart.so*"),
*get_library_path("nvidia.nvjitlink.lib", "libnvJitLink.so*"),
*get_library_path("nvidia.cublas.lib", "libcublasLt.so*"),
...

A related observation, which we have not traced further: the CUDA 13 builds of PyTorch (seen with torch==2.13.0+cu130) install their CUDA libraries into a single nvidia/cu13/lib directory, with no nvidia.cuda_runtime, nvidia.cusparse, ... packages. With such a torch, get_library_path matches none of the entries above, and nothing is preloaded.

Dominant language
Python
Stars
2.1k
Forks
663
Avg merge
3d 11h
Merged PRs (30d)
25

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepmodeling/deepmd-kit

All issues in deepmodeling/deepmd-kit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.