hapi_memory_daemon: EOF path spins at 1 kHz for the life of the run; EINTR path spins unthrottled

オープン
  • 何か問題がありますか？
#3,973 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
42/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
cmake, cpp, linux
領域
hpc, operating-systems, performance

調査の方向性

Start in src/arch/cuda/hybridAPI/hapi_memory_daemon.cpp at the FIFO open (O_RDONLY | O_NONBLOCK then blocking) and the main read() loop: the bytes_read == 0 branch with usleep(1000) and the EINTR continue. Cross-check launch and device init in hapi_impl.cpp (hapiStartMemoryDaemon, hapiSetDevice) and the SHRINKEXPAND / CMakeLists.txt build of the standalone daemon. Done means the idle path blocks instead of spinning, EINTR is throttled or handled via poll/epoll, and placement/pinning is documented for the external launcher.

索引モデルが issue の本文から書いたものです。

説明

Summary

src/arch/cuda/hybridAPI/hapi_memory_daemon.cpp (the shrink/expand GPU-memory daemon, built as a standalone executable when SHRINKEXPAND is on — CMakeLists.txt:819) has a main loop that spins at ~1000 wakeups/second whenever no client is attached, which is most of a run. A second path spins unthrottled. Found by inspection while investigating core interference in GPU runs; not yet measured, so the severity claim below is analytical rather than empirical.

The 1 kHz path

The server FIFO is opened O_RDONLY | O_NONBLOCK (hapi_memory_daemon.cpp:144) and then switched to blocking (:151-152). With a FIFO in that state, read() returns 0 immediately when no writer is attached rather than blocking. The loop's response is:

else if (bytes_read == 0)
{
  // A writer closed the connection. The FIFO is still open.
  // The next read() will block until a new writer connects.
  // A small sleep prevents a potential tight spin-loop on misconfiguration.
  usleep(1000);
}

The comment's premise does not hold: the next read() does not block, it returns 0 again. So the usleep(1000) does not prevent a spin loop, it throttles one to 1 kHz — permanently, from before the first client ever connects, and again after every client disconnects.

The unthrottled path

else // bytes_read < 0
{
  if (errno == EINTR) {
    continue;   // no sleep
  }
  ...
}

If signals arrive repeatedly (profilers, the GPU runtime, job-control), this is a bare spin with no throttle at all.

Why it matters beyond the wasted cycles

The daemon's placement is not controlled by Charm. hapiStartMemoryDaemon() in hapi_impl.cpp:236 only creates FIFOs and blocks waiting on /tmp/daemon_ready_<device> — the daemon process itself is launched externally by the job script, so it inherits the launcher's affinity mask, not a PE's. On a node where that mask overlaps the worker cores, a 1 kHz wakeup source lands on a busy-polling PE's core. One daemon exists per device (four on a typical 4-GPU node).

The daemon also calls hapiSetDevice (hapi_memory_daemon.cpp:134) and so holds its own GPU context, which brings its own runtime helper threads, inheriting the daemon's mask. See charmplusplus/charm#3972 for why that thread class matters.

Note also that under CMK_SHRINK_EXPAND this call replaces the normal device-init path (hapi_impl.cpp:210-215), so shrink/expand builds take a different hapiSetDevice route from everyone else.

Suggested fixes

  • Block properly instead of polling: either keep a write descriptor open on the FIFO in the daemon so read() blocks rather than seeing EOF, or use poll()/epoll on the read end and only then read().
  • Give the EINTR path the same throttle as the EOF path, or handle it via poll() above.
  • Document the daemon's expected placement (and pin it in the launcher), since Charm cannot control it from inside.

How to confirm

On a shrink/expand run: grep Cpus_allowed_list /proc/<daemon_pid>/status for overlap with worker cores, and check the daemon's nonvoluntary_ctxt_switches / a PE's runqueue wait in /proc/<tid>/schedstat on any shared core.

主要言語
C++
スター
241
フォーク
62
平均マージ
1日 1時間
マージ済み PR（30日）
121

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

charmplusplus/charm のほかの issue

charmplusplus/charm の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。