HAPI: +gpushm aborts in hapiInit (ipcHandleOpen, invalid argument) when each process sees only its own GPU

Abierto
  • ¿Algo no está bien?
#4,200 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
49/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp
Área
distributed-systems

Línea de trabajo

Start in src/arch/cuda/hybridAPI/hapi_impl.cpp around ipcHandleOpen (the report cites lines 1722–1745) and trace how peer device buffers are opened and indexed. Reproduce with one visible GPU per task using --gpus-per-task=1 and +gpushm, then check the manual's GPU messaging section. Done when the behavior matches one of the stated expected outcomes and the manual describes the GPU visibility requirement.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

+gpushm aborts in hapiInit when each process of a node can see only its own GPU, which is the shape Slurm's --gpus-per-task=1 produces.

Observed

NCSA Delta, A100 node, CUDA 13.2, charm reconverse-linux-x86_64 cuda (branch ckdyn-dev7 at 669707835, recharm base 16f27646b; the HAPI code is unchanged there), two processes on one node:

srun --mpi=pmix -n 2 -c 4 --gpus-per-task=1 ./ckdyntest +pe 2 +gpushm
...
HAPI> hapi IPC event pool size - 16 per PE, 16 per device
HAPI> Enabling P2P access between devices
Fatal hapi Error [1] invalid argument at .../src/arch/cuda/hybridAPI/hapi_impl.cpp:1739
------- Processor 0 Exiting: Called CmiAbort ------
Reason: Exit due to hapi error
[0]   #0  hapiErrorDie (libcharm.so)
[0]   #1  hapiInit (libcharm.so)

Both processes abort the same way (job 23677329 on gpua011; the same for a charm4py program). The same binaries with both GPUs visible to both tasks (srun -n 2 --gpus=2 --gpu-bind=none ... +gpushm) start and run; IPC device transfers then work.

Cause

ipcHandleOpen (hapi_impl.cpp, around line 1722-1745) loops over every other process on the physical node and, for each of that process's devices, calls hapiIpcOpenMemHandle on the handle of its device communication buffer. With one GPU per task, that buffer lives on a device this process cannot see (Slurm sets CUDA_VISIBLE_DEVICES per task), and cuIpcOpenMemHandle returns invalid argument. The index device_count * i + j also assumes every process on the node sees the same number of devices.

Expected

Either +gpushm skips peers whose devices are not visible and falls back to the non-IPC path for them, with a message, or hapiInit prints that +gpushm needs all of the node's GPUs visible to every process and aborts with that text instead of invalid argument. The manual's GPU messaging section should state the requirement either way.

Found while validating charm4py 2's device paths on Delta, 2026-11-08. Logs on Delta: /projects/mzu/lkale/software/charm4py-thin/devp-23677329/py_posted_receive_n2shm.log and ckdyn_n2shm.log.

🤖 Generated with Claude Code

Lenguaje dominante
C++
Estrellas
241
Forks
62
Merge medio
1 d 1 h
PR fusionados (30 d)
125

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de charmplusplus/charm

Todos los issues de charmplusplus/charm

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.