[Feature Request] Bind single-rank GPU runs to the NUMA node of their GPU

オープン
  • 何か問題がありますか？
#6,092 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
cpp, linux
領域
backend, performance

調査の方向性

Start by identifying where the LAMMPS CUDA device is initialized and how this project handles CPU affinity; the issue does not name source files or tests. Compare automatic NUMA binding with the documentation-only alternative, while preserving launcher and user affinity and room for OpenMP threads. Done means either safe opt-in or disableable binding for unbound single-rank runs, or a documented numactl recommendation.

索引モデルが issue の本文から書いたものです。

説明

enhancement
Summary

A single-rank LAMMPS run started directly, without mpirun, has no CPU binding. The scheduler may then place the host process on a NUMA node far from its GPU. For small and medium systems the host side matters, and binding the process to the GPU's NUMA node gives a small, measurable speedup. DeePMD-kit could do this binding automatically when the launcher has not set one, or at least recommend it in the documentation.

Detailed Description

Observation. Measured with compressed DPA4C on the LAMMPS CUDA path, on a workstation with 8 × RTX PRO 6000 and 192 CPUs. The NUMA node of GPU 0 holds CPUs 0–23 and 96–119. The numbers are paired medians of six alternating rounds.

  • Bound to one core versus a direct, unbound start: Mini at 10,296 atoms is 0.9% faster, and Nano at 512 atoms is 2.1% faster.
  • A direct start confined to the CPUs of the GPU's NUMA node is as fast as the bound run for Mini at 10,296 atoms (+0.2%). Confined to another NUMA node, it is 0.65% slower. So at this size the gain comes from host-to-GPU locality.
  • At 512 atoms the step is host bound, and pinning to a single core gains a further 2.5% beyond NUMA confinement.

Suggested approach. This follows the NUMA binding of DeepSeek's DeepInfer, which looks up the NUMA node of each accelerator, restricts the process's CPU affinity to that node through libnuma and binds memory allocation there.

  1. When the device is initialized, look up the NUMA node of the GPU, for example from /sys/bus/pci/devices/<bdf>/numa_node.
  2. Only when the process affinity is still the whole machine, meaning neither the launcher nor the user has bound it, restrict the affinity to the CPUs of that node and optionally set the memory policy.
  3. Provide an environment variable to disable the binding.

The binding must not override a binding chosen by mpirun, Slurm or numactl, and it must leave room for the OpenMP threads of LAMMPS. A simpler alternative is a short note in the LAMMPS installation and running documentation recommending numactl --cpunodebind=<node> --membind=<node> for single-rank GPU runs.

The expected gain is a few percent at most, mainly for small, host-bound systems.

主要言語
Python
スター
2.1k
フォーク
663
平均マージ
3日 11時間
マージ済み PR（30日）
25

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

deepmodeling/deepmd-kit のほかの issue

deepmodeling/deepmd-kit の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。