hapitest: host memory grows without bound before the first per-PE report (OOM-killed on Delta, CUDA 13.2)

Aperta
  • Qualcosa non va?
#4,197 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, testing-qa

Direzione di ricerca

Start with tests/charm++/cuda/hapitest, especially PeCheck::createStreams and PeCheck::check; the report places the unbounded host memory growth between the chare summary and Main::peReport. Run the tiny verbose command from the issue under the same CUDA build, then add progress prints after each check block or break on cudaMallocHost/cudaHostAlloc to identify the growing allocation. Done means the allocation source is identified and the test reaches its first per-PE report without unbounded host memory growth.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

tests/charm++/cuda/hapitest (the HAPI portable-core acceptance test, #3951) is killed for host memory exhaustion on NCSA Delta before it reaches its first per-PE report. The growth does not depend on the problem size, so it happens in the stream-pool creation or the per-PE checks, not in the kernel chains.

Environment

  • NCSA Delta, A40 nodes (gpuA40x4-interactive, sm_86), one node, RHEL 9, Slurm.
  • CUDA 13.2 (cudatoolkit/26.5_13.2), gcc 14.2.1, libfabric 2.3.1, system hwloc.
  • charm at 16f27646b (recharm) plus the ckdyn-dev commits, which touch nothing under src/arch/cuda or tests/charm++/cuda (git diff 16f27646b..6a57e5553 --stat -- src/arch/cuda tests/charm++/cuda is empty); reconverse submodule ee47fb6 (ba836e6 plus a CCS dispatch fix in conv-ccs.cpp, unrelated).
  • Build: ./build charm++ reconverse-linux-x86_64 cuda -j16 --with-production --disable-fortran --with-refnum-type="unsigned int" --with-cmake-args=-DLCI_OPTIMIZE_FOR_NATIVE=OFF.
  • Test built in the build tree with make CUDA_ARCH=sm_86.
  • Run inside an allocation with srun --mpi=pmix -n 1 -c 4 --gpus-per-task=1 ./hapitest ....

Observed

Default sizes, 32 GB job, one process:

HAPI> Config: 1 device(s) per process, 1 PE(s) per device, 1 device(s) per host
HAPI> Enabling P2P access between devices
[hapitest] PEs 1, logical nodes 1, physical nodes 1
[hapitest] chares 2, chains/chare 8, doubles/chain 1024
error: Detected 1 oom_kill event in StepId=23670313.6. Some of the step tasks have been OOM Killed.
srun: error: gpub061: task 0: Out Of Memory

sacct MaxRSS for that step: 33489084 KB after 27 s. Two processes (-n 2 ./hapitest +pe 2): same, killed after 15 s at 17.4 GB.

Tiny sizes, 8 GB step limit, verbose:

srun --mem=8g ... ./hapitest -v -c 2 -k 1 -n 64 +pe 1
[hapitest] chares 2, chains/chare 1, doubles/chain 64
error: Detected 1 oom_kill event in StepId=23670643.8 ...

MaxRSS 8388344 KB after 8 s. Nothing is printed after the "chares" line in either case, so the growth is in the code between that line and Main::peReport: PeCheck::createStreams (hapiCreateStreams() on rank 0) and PeCheck::check (device count and mapping checks, hapiGetStream, the four hapiMallocHost/hapiFreeHost spellings including the pooled ones, the buddy-allocator round trips). The kernel chains (Worker::run) are never reached, which is why -c, -k and -n make no difference. The reduction that createStreams contributes to never completes, or check never returns.

Not the cause

  • Not the CkDyn layer or charm4py: the same build runs tests/charm++/ckdyn (including its new device and HAPI-callback phases) and tests/charm++/cuda-style device transfers between processes and between nodes without memory growth.
  • Not the problem size (above).

Logs

Job output and per-step logs are on Delta under /projects/mzu/lkale/software/charm4py-thin/gate-23670313/ and gate-23670643/ (hapitest_n1.log, hapitest_n2.log, hapitest_small_n1.log), with copies on Kale's laptop under charm4py-thin/logs/step-3/delta/.

Suggested next step

Run hapitest under the same build with -v and a CkPrintf after each block of PeCheck::check, or under gdb with a breakpoint on cudaMallocHost/cudaHostAlloc, to see which call allocates without bound. The pooled hapiMallocHost(&h, nbytes, true) path and hapiCreateStreams() are the two calls in that window that touch HAPI's own pools.

Found while validating charm4py 2 (thin) on Delta, 2026-11-08.

🤖 Generated with Claude Code

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.