hapitest: host memory grows without bound before the first per-PE report (OOM-killed on Delta, CUDA 13.2)

Open
  • Is something wrong?
#4,197 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
43/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp
Domain
backend, testing-qa

Research direction

Start with tests/charm++/cuda/hapitest, especially PeCheck::createStreams and PeCheck::check; the report places the unbounded host memory growth between the chare summary and Main::peReport. Run the tiny verbose command from the issue under the same CUDA build, then add progress prints after each check block or break on cudaMallocHost/cudaHostAlloc to identify the growing allocation. Done means the allocation source is identified and the test reaches its first per-PE report without unbounded host memory growth.

Written by the indexing model from the issue text.

Description

tests/charm++/cuda/hapitest (the HAPI portable-core acceptance test, #3951) is killed for host memory exhaustion on NCSA Delta before it reaches its first per-PE report. The growth does not depend on the problem size, so it happens in the stream-pool creation or the per-PE checks, not in the kernel chains.

Environment

  • NCSA Delta, A40 nodes (gpuA40x4-interactive, sm_86), one node, RHEL 9, Slurm.
  • CUDA 13.2 (cudatoolkit/26.5_13.2), gcc 14.2.1, libfabric 2.3.1, system hwloc.
  • charm at 16f27646b (recharm) plus the ckdyn-dev commits, which touch nothing under src/arch/cuda or tests/charm++/cuda (git diff 16f27646b..6a57e5553 --stat -- src/arch/cuda tests/charm++/cuda is empty); reconverse submodule ee47fb6 (ba836e6 plus a CCS dispatch fix in conv-ccs.cpp, unrelated).
  • Build: ./build charm++ reconverse-linux-x86_64 cuda -j16 --with-production --disable-fortran --with-refnum-type="unsigned int" --with-cmake-args=-DLCI_OPTIMIZE_FOR_NATIVE=OFF.
  • Test built in the build tree with make CUDA_ARCH=sm_86.
  • Run inside an allocation with srun --mpi=pmix -n 1 -c 4 --gpus-per-task=1 ./hapitest ....

Observed

Default sizes, 32 GB job, one process:

HAPI> Config: 1 device(s) per process, 1 PE(s) per device, 1 device(s) per host
HAPI> Enabling P2P access between devices
[hapitest] PEs 1, logical nodes 1, physical nodes 1
[hapitest] chares 2, chains/chare 8, doubles/chain 1024
error: Detected 1 oom_kill event in StepId=23670313.6. Some of the step tasks have been OOM Killed.
srun: error: gpub061: task 0: Out Of Memory

sacct MaxRSS for that step: 33489084 KB after 27 s. Two processes (-n 2 ./hapitest +pe 2): same, killed after 15 s at 17.4 GB.

Tiny sizes, 8 GB step limit, verbose:

srun --mem=8g ... ./hapitest -v -c 2 -k 1 -n 64 +pe 1
[hapitest] chares 2, chains/chare 1, doubles/chain 64
error: Detected 1 oom_kill event in StepId=23670643.8 ...

MaxRSS 8388344 KB after 8 s. Nothing is printed after the "chares" line in either case, so the growth is in the code between that line and Main::peReport: PeCheck::createStreams (hapiCreateStreams() on rank 0) and PeCheck::check (device count and mapping checks, hapiGetStream, the four hapiMallocHost/hapiFreeHost spellings including the pooled ones, the buddy-allocator round trips). The kernel chains (Worker::run) are never reached, which is why -c, -k and -n make no difference. The reduction that createStreams contributes to never completes, or check never returns.

Not the cause

  • Not the CkDyn layer or charm4py: the same build runs tests/charm++/ckdyn (including its new device and HAPI-callback phases) and tests/charm++/cuda-style device transfers between processes and between nodes without memory growth.
  • Not the problem size (above).

Logs

Job output and per-step logs are on Delta under /projects/mzu/lkale/software/charm4py-thin/gate-23670313/ and gate-23670643/ (hapitest_n1.log, hapitest_n2.log, hapitest_small_n1.log), with copies on Kale's laptop under charm4py-thin/logs/step-3/delta/.

Suggested next step

Run hapitest under the same build with -v and a CkPrintf after each block of PeCheck::check, or under gdb with a breakpoint on cudaMallocHost/cudaHostAlloc, to see which call allocates without bound. The pooled hapiMallocHost(&h, nbytes, true) path and hapiCreateStreams() are the two calls in that window that touch HAPI's own pools.

Found while validating charm4py 2 (thin) on Delta, 2026-11-08.

🤖 Generated with Claude Code

Dominant language
C++
Stars
241
Forks
62
Avg merge
1d 1h
Merged PRs (30d)
121

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from charmplusplus/charm

All issues in charmplusplus/charm

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.