Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[BUG] lmp entry point does not preload nvJitLink, so the wheels' cuSPARSE and cuSOLVER can bind an older toolkit libnvJitLink

Abierto Apto para principiantes
  • ¿Algo no está bien?
#6,086 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
73/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
backend, devops

Línea de trabajo

Empieza con deepmd/lmp.py e inspecciona cómo se construye la lista LD_PRELOAD mediante get_library_path; el informe identifica la biblioteca nvidia.nvjitlink.lib que falta y el orden de carga requerido. Reproduce el problema con los comandos LD_BIND_NOW=1 proporcionados y el toolkit CUDA 12.8 en LD_LIBRARY_PATH. La tarea estará completa cuando precargar el nvJitLink del wheel antes de cuSOLVER y cuSPARSE evite el error de búsqueda de símbolos informado.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Bug summary

deepmd/lmp.py, which sets up the environment of the lmp entry point of deepmd-kit[lmp], puts the CUDA libraries of the NVIDIA wheels into LD_PRELOAD: cudart, cublasLt, cublas, cufft, curand, cusolver, cusparse and cudnn. libnvJitLink is missing from that list, although three of the preloaded wheel libraries have it as a DT_NEEDED dependency: libcusparse.so.12, libcusolver.so.11 and libcusolverMg.so.11.

The dynamic loader resolves that dependency through LD_LIBRARY_PATH before it tries the wheel library's own RUNPATH ($ORIGIN/../../nvjitlink/lib). Clusters often put a CUDA toolkit on LD_LIBRARY_PATH, for example through a cuda/12.x module. If that toolkit is older than the CUDA of the torch wheel, the preloaded cuSPARSE binds the toolkit's older libnvJitLink.so.12. That library lacks the versioned symbols of the newer release, so the process fails with a symbol lookup error.

DeePMD-kit Version

master (668b60cc2); deepmd/lmp.py is unchanged since #5707

Backend and its version

PyTorch 2.13.0+cu129 (pip, with the NVIDIA CUDA wheels); CUDA toolkit 12.8 on LD_LIBRARY_PATH; driver 535.161.08

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

We hit it while porting to an H20 node, in a LAMMPS built from source against the same pip torch (not the deepmd-kit[lmp] wheel), when DeePMD-kit loaded the PyTorch exportable backend for a .pt2 model:

Unable to load PyTorch Exportable backend plugin (libdeepmd_backend_ptexpt.so) ... nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12

We did not run the deepmd-kit[lmp] wheel itself; how it is affected follows from the preload list and the reproduction below. A Python process is not affected, because torch loads the libraries of its own wheels first.

Steps to Reproduce

SITE is the site-packages of an environment with torch==2.13.0+cu129 and its NVIDIA wheels; /usr/local/cuda is a CUDA 12.8 toolkit. Test 1 preloads cuSPARSE the way deepmd/lmp.py does, with the older toolkit on LD_LIBRARY_PATH:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# /bin/true: symbol lookup error: .../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12
# exit status 127

Test 2 preloads the wheel's nvJitLink first:

LD_BIND_NOW=1 LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
  LD_PRELOAD=$SITE/nvidia/nvjitlink/lib/libnvJitLink.so.12:$SITE/nvidia/cusparse/lib/libcusparse.so.12 /bin/true
# exit status 0

LD_BIND_NOW=1 makes the error appear at startup. Without it, the error appears when the symbols are first resolved; in our run, that was when the backend plugin was loaded.

The wheel libraries that need nvJitLink can be listed with readelf -d:

for f in $SITE/nvidia/*/lib/lib*.so.*; do readelf -d "$f" | grep -q 'NEEDED.*libnvJitLink' && echo "$f"; done
# .../nvidia/cusolver/lib/libcusolver.so.11
# .../nvidia/cusolver/lib/libcusolverMg.so.11
# .../nvidia/cusparse/lib/libcusparse.so.12
Further Information, Files, and Links

Suggested fix: preload the wheel's nvJitLink ahead of cuSOLVER and cuSPARSE. LD_PRELOAD loads its entries in order, and a library that is already loaded satisfies a later DT_NEEDED with the same soname:

*get_library_path("nvidia.cuda_runtime.lib", "libcudart.so*"),
*get_library_path("nvidia.nvjitlink.lib", "libnvJitLink.so*"),
*get_library_path("nvidia.cublas.lib", "libcublasLt.so*"),
...

A related observation, which we have not traced further: the CUDA 13 builds of PyTorch (seen with torch==2.13.0+cu130) install their CUDA libraries into a single nvidia/cu13/lib directory, with no nvidia.cuda_runtime, nvidia.cusparse, ... packages. With such a torch, get_library_path matches none of the entries above, and nothing is preloaded.

Lenguaje dominante
Python
Estrellas
2.1k
Forks
661
Merge medio
4 d 13 h
PR fusionados (30 d)
9

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de deepmodeling/deepmd-kit

Todos los issues de deepmodeling/deepmd-kit

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.