[Feature Request] Bind single-rank GPU runs to the NUMA node of their GPU

Abierto
  • ¿Algo no está bien?
#6,092 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp, linux
Área
backend, performance

Línea de trabajo

Start by identifying where the LAMMPS CUDA device is initialized and how this project handles CPU affinity; the issue does not name source files or tests. Compare automatic NUMA binding with the documentation-only alternative, while preserving launcher and user affinity and room for OpenMP threads. Done means either safe opt-in or disableable binding for unbound single-rank runs, or a documented numactl recommendation.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement
Summary

A single-rank LAMMPS run started directly, without mpirun, has no CPU binding. The scheduler may then place the host process on a NUMA node far from its GPU. For small and medium systems the host side matters, and binding the process to the GPU's NUMA node gives a small, measurable speedup. DeePMD-kit could do this binding automatically when the launcher has not set one, or at least recommend it in the documentation.

Detailed Description

Observation. Measured with compressed DPA4C on the LAMMPS CUDA path, on a workstation with 8 × RTX PRO 6000 and 192 CPUs. The NUMA node of GPU 0 holds CPUs 0–23 and 96–119. The numbers are paired medians of six alternating rounds.

  • Bound to one core versus a direct, unbound start: Mini at 10,296 atoms is 0.9% faster, and Nano at 512 atoms is 2.1% faster.
  • A direct start confined to the CPUs of the GPU's NUMA node is as fast as the bound run for Mini at 10,296 atoms (+0.2%). Confined to another NUMA node, it is 0.65% slower. So at this size the gain comes from host-to-GPU locality.
  • At 512 atoms the step is host bound, and pinning to a single core gains a further 2.5% beyond NUMA confinement.

Suggested approach. This follows the NUMA binding of DeepSeek's DeepInfer, which looks up the NUMA node of each accelerator, restricts the process's CPU affinity to that node through libnuma and binds memory allocation there.

  1. When the device is initialized, look up the NUMA node of the GPU, for example from /sys/bus/pci/devices/<bdf>/numa_node.
  2. Only when the process affinity is still the whole machine, meaning neither the launcher nor the user has bound it, restrict the affinity to the CPUs of that node and optionally set the memory policy.
  3. Provide an environment variable to disable the binding.

The binding must not override a binding chosen by mpirun, Slurm or numactl, and it must leave room for the OpenMP threads of LAMMPS. A simpler alternative is a short note in the LAMMPS installation and running documentation recommending numactl --cpunodebind=<node> --membind=<node> for single-rank GPU runs.

The expected gain is a few percent at most, mainly for small, host-bound systems.

Lenguaje dominante
Python
Estrellas
2.1k
Forks
663
Merge medio
3 d 11 h
PR fusionados (30 d)
25

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de deepmodeling/deepmd-kit

Todos los issues de deepmodeling/deepmd-kit

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.