Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[BUG] lmp entry point does not preload nvJitLink, so the wheels' cuSPARSE and cuSOLVER can bind an older toolkit libnvJitLink

Aperta Adatta ai principianti
  • Qualcosa non va?
#6,086 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
73/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, devops

Direzione di ricerca

Inizia da deepmd/lmp.py ed esamina come viene assemblata la lista LD_PRELOAD tramite get_library_path; il rapporto identifica la libreria mancante nvidia.nvjitlink.lib e l’ordine di caricamento richiesto. Riproduci il problema con i comandi LD_BIND_NOW=1 forniti e il toolkit CUDA 12.8 su LD_LIBRARY_PATH. Il lavoro è completato quando precaricare nvJitLink del wheel prima di cuSOLVER e cuSPARSE evita l’errore di ricerca del simbolo segnalato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug summary

deepmd/lmp.py, which sets up the environment of the lmp entry point of deepmd-kit[lmp], puts the CUDA libraries of the NVIDIA wheels into LD_PRELOAD: cudart, cublasLt, cublas, cufft, curand, cusolver, cusparse and cudnn. libnvJitLink is missing from that list, although three of the preloaded wheel libraries have it as a DT_NEEDED dependency: libcusparse.so.12, libcusolver.so.11 and libcusolverMg.so.11.

The dynamic loader resolves that dependency through LD_LIBRARY_PATH before it tries the wheel library's own RUNPATH ($ORIGIN/../../nvjitlink/lib). Clusters often put a CUDA toolkit on LD_LIBRARY_PATH, for example through a cuda/12.x module. If that toolkit is older than the CUDA of the torch wheel, the preloaded cuSPARSE binds the toolkit's older libnvJitLink.so.12. That library lacks the versioned symbols of the newer release, so the process fails with a symbol lookup error.

DeePMD-kit Version

master (668b60cc2); deepmd/lmp.py is unchanged since #5707

Backend and its version

PyTorch 2.13.0+cu129 (pip, with the NVIDIA CUDA wheels); CUDA toolkit 12.8 on LD_LIBRARY_PATH; driver 535.161.08

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

We hit it while porting to an H20 node, in a LAMMPS built from source against the same pip torch (not the deepmd-kit[lmp] wheel), when DeePMD-kit loaded the PyTorch exportable backend for a .pt2 model:

Unable to load PyTorch Exportable backend plugin (libdeepmd_backend_ptexpt.so) ... nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12

We did not run the deepmd-kit[lmp] wheel itself; how it is affected follows from the preload list and the reproduction below. A Python process is not affected, because torch loads the libraries of its own wheels first.

Steps to Reproduce

SITE is the site-packages of an environment with torch==2.13.0+cu129 and its NVIDIA wheels; /usr/local/cuda is a CUDA 12.8 toolkit. Test 1 preloads cuSPARSE the way deepmd/lmp.py does, with the older toolkit on LD_LIBRARY_PATH:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# /bin/true: symbol lookup error: .../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12
# exit status 127

Test 2 preloads the wheel's nvJitLink first:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/nvjitlink/lib/libnvJitLink.so.12:$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# exit status 0

LD_BIND_NOW=1 makes the error appear at startup. Without it, the error appears when the symbols are first resolved; in our run, that was when the backend plugin was loaded.

The wheel libraries that need nvJitLink can be listed with readelf -d:

for f in $SITE/nvidia/*/lib/lib*.so.*; do readelf -d "$f" | grep -q 'NEEDED.*libnvJitLink' && echo "$f"; done
# .../nvidia/cusolver/lib/libcusolver.so.11
# .../nvidia/cusolver/lib/libcusolverMg.so.11
# .../nvidia/cusparse/lib/libcusparse.so.12
Further Information, Files, and Links

Suggested fix: preload the wheel's nvJitLink ahead of cuSOLVER and cuSPARSE. LD_PRELOAD loads its entries in order, and a library that is already loaded satisfies a later DT_NEEDED with the same soname:

*get_library_path("nvidia.cuda_runtime.lib", "libcudart.so*"),
*get_library_path("nvidia.nvjitlink.lib", "libnvJitLink.so*"),
*get_library_path("nvidia.cublas.lib", "libcublasLt.so*"),
...

A related observation, which we have not traced further: the CUDA 13 builds of PyTorch (seen with torch==2.13.0+cu130) install their CUDA libraries into a single nvidia/cu13/lib directory, with no nvidia.cuda_runtime, nvidia.cusparse, ... packages. With such a torch, get_library_path matches none of the entries above, and nothing is preloaded.

Lingua principale
Python
Stelle
2.1k
Fork
659
Merge medio
5g 1h
PR unite (30g)
8

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di deepmodeling/deepmd-kit

Tutte le issue di deepmodeling/deepmd-kit

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.