GPU D2D: sender blocks on hapiStreamSynchronize in the intra-process path

Aperta
  • Qualcosa non va?
#3,957 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
28/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
cpp
Ambito
hpc

Direzione di ricerca

Start from CkRdmaDeviceOnSender on the MEMCPY intra-process path, where each buffer’s hapi_stream is synchronized before the metadata message is sent. Trace how charmxi-generated code issues that send immediately after the function returns, and how other HAPI completions use events plus hapiPollEvents instead of blocking. Done means the metadata send is deferred to an event callback without stalling the PE, with message ordering after the send still defined. This is a runtime design change, not a one-file edit; review #3955 and the overdecomposition notes first.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

CkRdmaDeviceOnSender synchronises each buffer's stream before the metadata message goes out, on the MEMCPY (same-process) path:

if (transfer_mode == CkNcpyModeDevice::MEMCPY) {
  // The receiver dereferences buffers[i]->ptr directly, so the producing
  // kernel/copy has to have retired before the metadata message goes out.
  for (int i = 0; i < numops; i++)
    hapiCheck(hapiStreamSynchronize(buffers[i]->hapi_stream));
  return;
}

It is correct — the receiver reads the pointer directly, so the producing work must have retired — but it blocks the PE until the GPU finishes, which is at odds with how the rest of the runtime treats device work. Every other completion in HAPI is asynchronous: an event is recorded and hapiPollEvents delivers a Charm++ callback from the scheduler loop. Here the PE simply waits.

The cost lands exactly where overdecomposition is supposed to help. With several chares per PE, one chare's send stalls the others' entry methods for the duration of a kernel, so the intra-process path — the one that should be cheapest — becomes the one that serialises a PE.

Why this is not a quick fix. Deferring it means the metadata message itself has to be sent asynchronously: record an event, and send from the completion callback rather than from CkRdmaDeviceOnSender. The send is issued by charmxi-generated code immediately after CkRdmaDeviceOnSender returns, so the generated path has to learn to defer, and ordering guarantees for messages issued after it need thought. That is a design change, not a local edit — which is why it was left alone in #3955.

Related: the same reasoning appears in the overdecomposition design notes: the runtime should hold work in software where it can be reordered, rather than blocking. A deferred-send mechanism here is a small instance of that.

Found during review of #3955 (GPU stage 9.2). Not a defect in that PR — the behaviour predates it — and deliberately postponed after discussion.

🤖 Generated with Claude Code

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.