[Feature Request] Bind single-rank GPU runs to the NUMA node of their GPU

Open
  • Is something wrong?
#6,092 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, linux
Domain
backend, performance

Research direction

Start by identifying where the LAMMPS CUDA device is initialized and how this project handles CPU affinity; the issue does not name source files or tests. Compare automatic NUMA binding with the documentation-only alternative, while preserving launcher and user affinity and room for OpenMP threads. Done means either safe opt-in or disableable binding for unbound single-rank runs, or a documented numactl recommendation.

Written by the indexing model from the issue text.

Description

enhancement
Summary

A single-rank LAMMPS run started directly, without mpirun, has no CPU binding. The scheduler may then place the host process on a NUMA node far from its GPU. For small and medium systems the host side matters, and binding the process to the GPU's NUMA node gives a small, measurable speedup. DeePMD-kit could do this binding automatically when the launcher has not set one, or at least recommend it in the documentation.

Detailed Description

Observation. Measured with compressed DPA4C on the LAMMPS CUDA path, on a workstation with 8 × RTX PRO 6000 and 192 CPUs. The NUMA node of GPU 0 holds CPUs 0–23 and 96–119. The numbers are paired medians of six alternating rounds.

  • Bound to one core versus a direct, unbound start: Mini at 10,296 atoms is 0.9% faster, and Nano at 512 atoms is 2.1% faster.
  • A direct start confined to the CPUs of the GPU's NUMA node is as fast as the bound run for Mini at 10,296 atoms (+0.2%). Confined to another NUMA node, it is 0.65% slower. So at this size the gain comes from host-to-GPU locality.
  • At 512 atoms the step is host bound, and pinning to a single core gains a further 2.5% beyond NUMA confinement.

Suggested approach. This follows the NUMA binding of DeepSeek's DeepInfer, which looks up the NUMA node of each accelerator, restricts the process's CPU affinity to that node through libnuma and binds memory allocation there.

  1. When the device is initialized, look up the NUMA node of the GPU, for example from /sys/bus/pci/devices/<bdf>/numa_node.
  2. Only when the process affinity is still the whole machine, meaning neither the launcher nor the user has bound it, restrict the affinity to the CPUs of that node and optionally set the memory policy.
  3. Provide an environment variable to disable the binding.

The binding must not override a binding chosen by mpirun, Slurm or numactl, and it must leave room for the OpenMP threads of LAMMPS. A simpler alternative is a short note in the LAMMPS installation and running documentation recommending numactl --cpunodebind=<node> --membind=<node> for single-rank GPU runs.

The expected gain is a few percent at most, mainly for small, host-bound systems.

Dominant language
Python
Stars
2.1k
Forks
663
Avg merge
3d 11h
Merged PRs (30d)
25

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepmodeling/deepmd-kit

All issues in deepmodeling/deepmd-kit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.