AMPI: megampi under +randomized_msgq aborts with "mis-matched client callbacks in reduction messages" after migrations

Aperta
  • Qualcosa non va?
#4,139 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
distributed-systems

Direzione di ricerca

Start with CkReductionMgr::reduceMessages and the AMPI barrier path reached from MPI_Win_create; inspect how reduction contributions are matched after ranks migrate and messages from one sender arrive out of order. Reproduce on Linux with the tests/ampi/megampi commands in the issue, enabling +randomized_msgq and repeating seeds. Done means the migrated run no longer aborts with mismatched client callbacks across the reported scenarios.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

AMPI

What happens

tests/ampi/megampi run under +randomized_msgq sometimes aborts with

mis-matched client callbacks in reduction messages

from CkReductionMgr::reduceMessages, reached from ampi::barrier() inside MPI_Win_create, right after the ranks have migrated (megampi migrates its ranks between its test phases). It does not happen on every seed, and it is not a start-up problem: the AMPI start-up ordering fault is fixed in #4137, and this abort was seen while testing that fix (reported in the "Not addressed" list of #4137's description).

Where to look

The reduction manager checks that every contribution of one reduction carries the same client callback. AMPI's barrier contributes with a callback, and after a migration the ranks on a PE may contribute to different reduction numbers than the manager expects when messages from one sender reach a PE out of order. +randomized_msgq makes that inversion routine; in reconverse it also happens without the flag when several PE threads of a process poll the network. #4032 (CkMulticast) and #4137 (AMPI start-up) were the same kind of hidden ordering dependence.

How to reproduce

Needs Linux: on macOS a thread cannot migrate between processes with address space layout randomization on, so megampi's multi-process runs are skipped there.

./build AMPI reconverse-linux-x86_64 -g -j8
cd tests/ampi/megampi && make
setarch $(uname -m) -R ./pgm +pe 4 +vp8 +randomized_msgq        # single process
lcrun -n 2 setarch $(uname -m) -R ./pgm +pe 4 +vp8 +randomized_msgq   # two processes

Repeat with several seeds (+randomized_msgq_seed N if the build has it, otherwise repeated runs) until the abort shows.

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
125

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.