hapi_memory_daemon: EOF path spins at 1 kHz for the life of the run; EINTR path spins unthrottled

Aperta
  • Qualcosa non va?
#3,973 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cmake, cpp, linux
Ambito
hpc, operating-systems, performance

Direzione di ricerca

Start in src/arch/cuda/hybridAPI/hapi_memory_daemon.cpp at the FIFO open (O_RDONLY | O_NONBLOCK then blocking) and the main read() loop: the bytes_read == 0 branch with usleep(1000) and the EINTR continue. Cross-check launch and device init in hapi_impl.cpp (hapiStartMemoryDaemon, hapiSetDevice) and the SHRINKEXPAND / CMakeLists.txt build of the standalone daemon. Done means the idle path blocks instead of spinning, EINTR is throttled or handled via poll/epoll, and placement/pinning is documented for the external launcher.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

src/arch/cuda/hybridAPI/hapi_memory_daemon.cpp (the shrink/expand GPU-memory daemon, built as a standalone executable when SHRINKEXPAND is on — CMakeLists.txt:819) has a main loop that spins at ~1000 wakeups/second whenever no client is attached, which is most of a run. A second path spins unthrottled. Found by inspection while investigating core interference in GPU runs; not yet measured, so the severity claim below is analytical rather than empirical.

The 1 kHz path

The server FIFO is opened O_RDONLY | O_NONBLOCK (hapi_memory_daemon.cpp:144) and then switched to blocking (:151-152). With a FIFO in that state, read() returns 0 immediately when no writer is attached rather than blocking. The loop's response is:

else if (bytes_read == 0)
{
  // A writer closed the connection. The FIFO is still open.
  // The next read() will block until a new writer connects.
  // A small sleep prevents a potential tight spin-loop on misconfiguration.
  usleep(1000);
}

The comment's premise does not hold: the next read() does not block, it returns 0 again. So the usleep(1000) does not prevent a spin loop, it throttles one to 1 kHz — permanently, from before the first client ever connects, and again after every client disconnects.

The unthrottled path

else // bytes_read < 0
{
  if (errno == EINTR) {
    continue;   // no sleep
  }
  ...
}

If signals arrive repeatedly (profilers, the GPU runtime, job-control), this is a bare spin with no throttle at all.

Why it matters beyond the wasted cycles

The daemon's placement is not controlled by Charm. hapiStartMemoryDaemon() in hapi_impl.cpp:236 only creates FIFOs and blocks waiting on /tmp/daemon_ready_<device> — the daemon process itself is launched externally by the job script, so it inherits the launcher's affinity mask, not a PE's. On a node where that mask overlaps the worker cores, a 1 kHz wakeup source lands on a busy-polling PE's core. One daemon exists per device (four on a typical 4-GPU node).

The daemon also calls hapiSetDevice (hapi_memory_daemon.cpp:134) and so holds its own GPU context, which brings its own runtime helper threads, inheriting the daemon's mask. See charmplusplus/charm#3972 for why that thread class matters.

Note also that under CMK_SHRINK_EXPAND this call replaces the normal device-init path (hapi_impl.cpp:210-215), so shrink/expand builds take a different hapiSetDevice route from everyone else.

Suggested fixes

  • Block properly instead of polling: either keep a write descriptor open on the FIFO in the daemon so read() blocks rather than seeing EOF, or use poll()/epoll on the read end and only then read().
  • Give the EINTR path the same throttle as the EOF path, or handle it via poll() above.
  • Document the daemon's expected placement (and pin it in the launcher), since Charm cannot control it from inside.

How to confirm

On a shrink/expand run: grep Cpus_allowed_list /proc/<daemon_pid>/status for overlap with worker cores, and check the daemon's nonvoluntary_ctxt_switches / a PE's runqueue wait in /proc/<tid>/schedstat on any shared core.

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.