Charm++ messages are enqueued twice on the receiving PE: switch the handler before the first enqueue (and reconverse's CldHandler drops priorities)

Aperta
  • Qualcosa non va?
#4,022 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
distributed-systems

Direzione di ricerca

Start with cldb.none.cpp and ck.C, especially CldHandler, CldEnqueue, and _skipCldHandler; compare the reconverse and classic paths described in the issue. The issue marks this analysis-only and deferred: first address the chare-message correctness and priority behavior, then measure the group/array changes with Task Bench and ping_ack. Done means eliminating redundant queueing without losing priority or ordering guarantees.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

A Charm++ message that reaches its destination PE through the network is queued twice before its entry method runs, and the second queueing exists only because the first one is done before the handler has been switched. The goal of this issue: when a message is enqueued on the destination PE, its handler is already the one that runs the entry method, so it is never enqueued again into another queue.

The paths today (charm reviewed-with-reconverse, reconverse main)

Arrival: the network completion pushes the message to the destination PE's queue (CmiPushPE, the PCq). The scheduler pops it and calls the handler stored in the message. For a non-expedited message that handler is one of two enqueue-again handlers chosen on the send side:

sent by send side handler on the wire on arrival queue passes
singleton chare (CkSendMsg), Cld broadcasts _CldEnqueue → CldEnqueue CldHandlerIndex reconverse CldHandler (cldb.none.cpp): restores _charmHandlerIdx, computes queueing/priority via _infoFn, then CmiPushPE(CmiMyRank(), ...) -- back into the same PCq, priority discarded PCq, PCq
group (_sendMsgBranch), array element (CkArrayManagerDeliver) _skipCldEnqueue (ck.C) index_skipCldHandler, _charmHandlerIdx saved as XHandler _skipCldHandler (ck.C): restores _charmHandlerIdx, CqsEnqueueGeneral into CsdSchedQueue with the message's strategy and priority PCq, priority queue
expedited _noCldEnqueue _charmHandlerIdx runs _processHandler directly PCq

Only after the second dequeue does _processHandler dispatch by message type to _invokeEntry(epIdx, ...).

Problems
  1. Chare messages on reconverse: the second pass is pure overhead and loses priorities. Classic's cldb.none.C CldHandler does CsdEnqueueGeneral(msg, queueing, priobits, prioptr). Reconverse's does CmiPushPE(CmiMyRank(), len, msg) with the comment "use priority queue when we add priority queue"; the priority queue exists since reconverse #126 (2025-10). The same applies to CldEnqueue's local branch (pe == CmiMyPe()). So a chare entry method with a priority runs FIFO on reconverse, and every remote chare message goes through the PCq twice. One-line fix each.
  2. Groups and arrays: the second pass is the priority mechanism. It is needed only when a priority can reorder the message. In classic SMP the first pass is unavoidable (the comm thread may not touch a worker's CsdSchedQueue). The second pass is redundant when the message is FIFO with no priority bits and the PE's priority queue is empty: nothing can be ordered ahead of it, and same-sender order is preserved because both arrive through the PCq in order. _skipCldHandler could run _processHandler directly in that case.
  3. On reconverse the first pass can go too. The thread that polls the network is a worker PE, and with destination-device sends (under study in reconverse) it is usually the destination itself. The completion callback can then enqueue straight into the PE's own priority queue with _charmHandlerIdx set -- one queue pass, in the queue the message is dequeued from for execution. (Running the entry method inside the callback is not safe: sends made under LCI's progress callback go to its backlog.)
Not the point

Saving the switch in _processHandler or the two checks before it is tens of nanoseconds; the cost is the extra queue pass (a full self-send-schedule-handle cycle measures about 310 ns on a laptop).

Status

Analysis only; deferred. Deal with 1 first (correctness), then 2 and 3 with Task Bench / ping_ack measurements.

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.