AMPI: MPI_COMM_SELF element not found on the new PE when a rank resumes in AMPI_Migrate (cause unknown)

Abierto
  • ¿Algo no está bien?
#4,155 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
distributed-systems

Línea de trabajo

Start with removeUnimportantArrayObjsfromPeCache in ampi.C and the migration path through CkLocMgr::pupElementsFor in src/ck-core/cklocation.C. Reproduce with the examples/ampi/Cjacobi3D command recorded in the issue, then add the suggested diagnostic for manager count and the MPI_COMM_SELF array’s element table when lookup returns null. Done means explaining why the element can be absent when the rank resumes and identifying a reproducible fix; the cause is currently unknown.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What

In removeUnimportantArrayObjsfromPeCache (ampi.C), called from AMPI_Migrate when a rank resumes on a new PE after a load balancing step, the lookup of the rank's MPI_COMM_SELF element (comm2ampi(MPI_COMM_SELF), that is proxy[thisIndex].ckLocal()) can answer null. #4154 makes the pruning skip the element in that case, as #4075 intended; this issue records that the absence itself is not explained.

Where it was seen

The ampi-linux-x86_64 CI job of #4152, run 42065030423, first attempt, step "examples/ampi (2 processes, lcrun)": setarch x86_64 -R lcrun -n 2 ./jacobi.pup +pe 4 2 2 2 40 -chkpt 20 log.pup +vp8 +balancer RotateLB in examples/ampi/Cjacobi3D. PE 2 aborted with "AMPI's getAmpiInstance> null pointer", stack AMPI_Migrate → removeUnimportantArrayObjsfromPeCache → getAmpiInstance. The rerun passed, the job had passed its previous ten runs, and #4152 touches nothing on this path. On Anvil (debug build, two processes, setarch -R) the same command passed 40 of 40 on recharm 09c7dba1a. One process on macOS: 18 of 18.

What is known

  • The example balances at iterations 5, 15, 25 and 35, so the first migration happens well after MPI_Init; this is not a start-up race in array creation.
  • With +pe 4 in two processes, RotateLB moves every rank to PE+1 mod 4, so PE 1 → PE 2 and PE 3 → PE 0 cross processes; those migrations read the isomalloc heap through zero-copy pup (pup_buffer), and ckJustMigrated, which wakes the thread, runs only after the reads complete (CkLocMgr::immigrateWithDevice, processAfterActiveRgetsCompleted). The abort was on PE 2, so on a cross-process arrival.
  • All elements bound to the TCharm element, the ampiParent and one ampi element per communicator, are unpacked in CkLocMgr::pupElementsFor before any ckJustMigrated runs, so when the thread resumes every element should be in its array manager's table on the new PE.
  • pupElementsFor matches the source's array managers to the destination's by count (msg->nManagers against managers.size(), CkLocMgr::immigrate) and unpacks in the destination's manager order. A destination with the same number of managers but a different set would attach the elements to the wrong arrays. Not confirmed; by iteration 5 every PE should have every manager.

Why it matters

If the element can be absent when the thread resumes, a real MPI call on MPI_COMM_SELF right after the migration would abort the same way; that has not been seen. Reproducing needs many two-process runs on Linux with a diagnostic at the lookup (print the manager count and the element table of the MPI_COMM_SELF array when the lookup answers null).

Touches the charm++ core: possibly (src/ck-core/cklocation.C), if the manager-set hypothesis holds.

Lenguaje dominante
C++
Estrellas
241
Forks
62
Merge medio
1 d 1 h
PR fusionados (30 d)
125

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de charmplusplus/charm

Todos los issues de charmplusplus/charm

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.