AMPI: MPI_COMM_SELF element not found on the new PE when a rank resumes in AMPI_Migrate (cause unknown)

Aperta
  • Qualcosa non va?
#4,155 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
distributed-systems

Direzione di ricerca

Start with removeUnimportantArrayObjsfromPeCache in ampi.C and the migration path through CkLocMgr::pupElementsFor in src/ck-core/cklocation.C. Reproduce with the examples/ampi/Cjacobi3D command recorded in the issue, then add the suggested diagnostic for manager count and the MPI_COMM_SELF array’s element table when lookup returns null. Done means explaining why the element can be absent when the rank resumes and identifying a reproducible fix; the cause is currently unknown.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What

In removeUnimportantArrayObjsfromPeCache (ampi.C), called from AMPI_Migrate when a rank resumes on a new PE after a load balancing step, the lookup of the rank's MPI_COMM_SELF element (comm2ampi(MPI_COMM_SELF), that is proxy[thisIndex].ckLocal()) can answer null. #4154 makes the pruning skip the element in that case, as #4075 intended; this issue records that the absence itself is not explained.

Where it was seen

The ampi-linux-x86_64 CI job of #4152, run 42065030423, first attempt, step "examples/ampi (2 processes, lcrun)": setarch x86_64 -R lcrun -n 2 ./jacobi.pup +pe 4 2 2 2 40 -chkpt 20 log.pup +vp8 +balancer RotateLB in examples/ampi/Cjacobi3D. PE 2 aborted with "AMPI's getAmpiInstance> null pointer", stack AMPI_Migrate → removeUnimportantArrayObjsfromPeCache → getAmpiInstance. The rerun passed, the job had passed its previous ten runs, and #4152 touches nothing on this path. On Anvil (debug build, two processes, setarch -R) the same command passed 40 of 40 on recharm 09c7dba1a. One process on macOS: 18 of 18.

What is known

  • The example balances at iterations 5, 15, 25 and 35, so the first migration happens well after MPI_Init; this is not a start-up race in array creation.
  • With +pe 4 in two processes, RotateLB moves every rank to PE+1 mod 4, so PE 1 → PE 2 and PE 3 → PE 0 cross processes; those migrations read the isomalloc heap through zero-copy pup (pup_buffer), and ckJustMigrated, which wakes the thread, runs only after the reads complete (CkLocMgr::immigrateWithDevice, processAfterActiveRgetsCompleted). The abort was on PE 2, so on a cross-process arrival.
  • All elements bound to the TCharm element, the ampiParent and one ampi element per communicator, are unpacked in CkLocMgr::pupElementsFor before any ckJustMigrated runs, so when the thread resumes every element should be in its array manager's table on the new PE.
  • pupElementsFor matches the source's array managers to the destination's by count (msg->nManagers against managers.size(), CkLocMgr::immigrate) and unpacks in the destination's manager order. A destination with the same number of managers but a different set would attach the elements to the wrong arrays. Not confirmed; by iteration 5 every PE should have every manager.

Why it matters

If the element can be absent when the thread resumes, a real MPI call on MPI_COMM_SELF right after the migration would abort the same way; that has not been seen. Reproducing needs many two-process runs on Linux with a diagnostic at the lookup (print the manager count and the element table of the MPI_COMM_SELF array when the lookup answers null).

Touches the charm++ core: possibly (src/ck-core/cklocation.C), if the manager-set hypothesis holds.

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.