Device zerocopy examples repack send buffers with nothing guaranteeing the neighbour's get has finished reading them

Aperta
  • Qualcosa non va?
#3,963 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@adityapb ci sta già lavorando.

Dal 18/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Device zerocopy examples repack send buffers with nothing guaranteeing the neighbour's get has finished reading them

Every .C under examples/charm++/cuda/gpudirect and benchmarks/charm++/cuda/gpudirect constructs its send buffers as CkDeviceBuffer(ptr, stream) and none passes a source callback, although CkDeviceBuffer accepts one. jacobi2d (and the copies derived from it) repacks the same d_send_* buffers every iteration.

On the inter-node path the receiver reads the sender's buffer with an RDMA get issued when the metadata message is processed. The sender proceeds to its next iteration as soon as it has received its own ghosts, which does not depend on whether every neighbour's get of its buffer has completed, so the repack can overwrite a buffer a neighbour is still reading. Under 32 chares per GPU with lagging chares the window is real; the examples check no values, so it shows as nothing.

Two fixes, either sufficient:

  • Double-buffer the send sets, alternating by iteration parity. A neighbour that has sent its message i+1 has finished its get of buffer i, so no acknowledgement is needed. About ten lines; measured cost nil.
  • Request the source callback and gate the repack on it. With piggybacked acks (#3961) this costs 3-4% of a chare-iteration on a Jacobi stencil at 2-16 KB ghosts; standalone, 6-20%.

Reproduction and both fixes are in the jacobi-overdecomp benchmark on the jacobi-overdecomp-bench branch (-a requests callbacks with double-buffered sets).

A registered device pool (#3962) makes this more important: it removes the per-message release, so the source callback becomes the only signal a sender gets, and "the pool owns the lifetime" must not be read as "the buffer is safe to overwrite".

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.