Device zerocopy source callbacks: no way to tell which buffer completed, and no per-send completion

Aperta
  • Qualcosa non va?
#3,966 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
distributed-systems, hpc

Direzione di ricerca

This is API design for Charm++ device zerocopy source callbacks, not a first-timer patch. Read the issue’s two problems (empty callback vs refnum; per-buffer vs all-done) and related #3961/#3963. Entry points named in the text include CkDeviceBuffer, deviceMintAck, DevicePendingAck, deviceDrainAcksInto, and the host path invokeSourceCallback/constructSourceBufferObject. Done means maintainers pick an option (document refnum, diagnose, payload, and/or coalesced send-complete callback) and implement it in the runtime; per-buffer completion must remain.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Two rough edges in the device zerocopy source-callback interface, raised while reviewing #3961. Neither is a defect in that PR — both predate it — but applications are about to start building habits around this API, so they are worth settling now.

1. The callback does not say which buffer completed

A device source callback is delivered as an empty message. It reports that a buffer is free, not which one. With several buffers in flight the sender has to distinguish them itself, by reference number:

CkCallback cb(CkIndex_Block::ghostFree(NULL), thisProxy[thisIndex]);
cb.setRefnum(LEFT);                    // the user sets it, before handing the callback over
... CkDeviceBuffer(d_send_left_ghost, cb, comm_stream) ...

entry void ghostFree(AckMsg*);         // must take a message, or the closure carries no refnum
when ghostFree[LEFT](AckMsg*) { ... }

This is arguably the better mechanism — the refnum rides in an envelope that was being sent anyway, where the host path instead builds and delivers a whole CkNcpyBuffer (invokeSourceCallback → constructSourceBufferObject) so the receiver knows which buffer completed.

The problem is that nothing announces it, and getting it wrong fails silently. A zero-argument entry method carries no reference number, so when ghostFree[LEFT]() never matches and the program hangs with no diagnostic and no error. That has already cost two jobs to diagnose once.

Options, roughly in increasing cost:

  • document the refnum idiom as the supported way;
  • diagnose it — a refnum-bearing callback whose target entry method takes no message is always a bug, and could be caught at runtime or by charmxi;
  • deliver an identifying payload as the host path does (consistent with host semantics, but heavier than a refnum).

2. There is no "all buffers done" callback

Callbacks are minted per buffer — deviceMintAck is called once per CkDeviceBuffer in the sender loop — so an entry method carrying four zerocopy buffers produces four callbacks. An application that only wants to know "the send is finished" has to count them or thread them through SDAG.

Worth noting what is and is not already shared: every ack id from one send already rides back in a single piggybacked block (deviceDrainAcksInto(dest_pe, buffers[0])), so the wire cost is one block regardless. What is still one-for-one is delivery — the sender resolves each id and sends each callback separately.

Coalescing is therefore cheap, with one gap: DevicePendingAck records the pointer, size, registration handle and callback, but not which send an id came from, so there is nothing to group by today. Two shapes:

  • record a send id on each pending ack and fire one callback when the last id of a send resolves; or
  • let the API take one callback for the whole invocation, which gives the user the semantics directly rather than making the runtime infer them.

Per-buffer completion should stay available rather than be replaced. A buffer whose transfer finishes early can be reused early, and that is occasionally what an application wants. The point is that "all done" is the common case and should be the easy thing to ask for.

Context

Background and the full registration/reuse picture are written up separately; the relevant part here is that the source callback is the zerocopy analogue of ownership transfer. With a message you hand the buffer over at send and "when may I reuse it?" is moot; with zerocopy the application keeps the buffer, so the callback is the runtime's answer to a question that is genuinely live. Related: #3963 (shipped examples repack send buffers with no reuse guarantee).

🤖 Generated with Claude Code

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.