CUDA build: host zerocopy acks routed to the device handler (setNcpyOpInfo declared with 24 parameters, defined with 25)

Aperta
  • Qualcosa non va?
#3,983 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@adityapb ci sta già lavorando.

Dal 18/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

On a CUDA-enabled charm-on-reconverse build, four host-side zerocopy tests fail with 2 processes (1 node or 2): zerocopy_with_qd and bcast_nonzero_root hang under a flood of CkRdmaDevice> WARNING: ack for unknown id, dereg_and_nodereg segfaults in CkRdmaDeviceRecvHandler, zc_post_async aborts with "Called send on corrupted callback". All pass single-process, and all pass on a non-CUDA build (Delta jobs 22108646/22108647 versus 22110164/22110165). Found 2026-09-15 while rerunning the site tier on Delta (#3982).

Mechanism

Charm++ declares setNcpyOpInfo itself, in the CMK_RECONVERSE branch of src/util/cmirdmautils.h, with 24 parameters. Reconverse's definition (contrib/reconverse/src/cmirdmautils.cpp, declared in include/cmirdmautils.h) has 25: a trailing void *deviceRdmaOpInfo, stored into NcpyOperationInfo. Both are extern "C", so the call in performEmApiRget (src/ck-core/ckrdma.C) links against the wrong arity and the callee reads its last argument from a stack slot the caller never wrote. Every entry-method zerocopy operation therefore carries stack garbage in deviceRdmaOpInfo.

On builds with CMK_CUDA || CMK_HIP, CkRdmaDirectAckHandler (ckrdma.C:311) treats a non-null deviceRdmaOpInfo as "this completion belongs to a device transfer" and routes the ack to CkRdmaDeviceRecvHandler, which casts the pointer to a DeviceRdmaOp. A bogus tag gives the unknown-id flood; the device handler does its own QD accounting, so the host operation's QdProcess never runs (the hangs); dereferencing gives the segfault and the corrupted callback. Confirmed with a diagnostic print (Delta job 22110118): opMode=1 (CMK_EM_API) acks arriving in the device branch with deviceRdmaOpInfo pointing into the caller's stack.

Fix

PR #3984 passes the argument (guarded by CMK_RECONVERSE, since classic's signature keeps 24 parameters). With it, the four tests and the rest of tests/charm++/zerocopy pass with 2 processes on the CUDA build (job 22110493).

Follow-ups this exposes

  1. The declaration duplication is the defect class. Charm++ carries its own copies of reconverse declarations (src/util/cmirdmautils.h, and the CmiNcpyBuffer/NcpyOperationInfo mirrors) instead of including reconverse's headers. An extern "C" signature can drift without any compiler diagnostic, as it did here. Charm++ on reconverse should include contrib/reconverse/include/cmirdmautils.h and drop the copy.
  2. Use opMode, not a pointer sentinel, to tell device completions from host ones. NcpyOperationInfo::opMode is always written by setNcpyOpInfo (values 0-8 used; 9-255 free), so a CMK_DEVICE_API mode cannot be stack garbage. That flips the invariant from "every host path must remember to null a field" to "the device path must set its mode", which fails loudly on a path the GPU tests exercise instead of intermittently on host paths. setReverseModeForNcpyOpInfo needs a no-op case for it. deviceRdmaOpInfo stays as payload.
  3. Hygiene: reconverse's CmiNcpyBuffer default constructor leaves deviceRdmaOpInfo uninitialized (include/conv-rdma.h:206); the two explicit constructors set it. Reachable through CmiDeviceBuffer::lci_ncpy_buffer, pup_util.C:191, and reconverse's own tests/rdma_ping. One-line initializer; patch held for the reconverse side.
  4. No CI runs any test on a CUDA build. reconverse-cuda.yaml compiles only. Host tests on a CUDA build are a configuration NAMD's GPU build on reconverse uses; a GPU runner or a manual site-run step for CUDA builds would have caught this.
  5. Separate, pre-existing: tests/charm++/zerocopy/zc_post_modify_size with 2 processes segfaults on both the fixed and unfixed CUDA builds (jobs 22110662, 22110700), and it is not in the CI tier; likely the same defect as reconverse#223.
  6. A CUDA-enabled build cannot run on a node without a GPU at all: LCI's accelerator layer calls cuInit at startup and aborts. Worth its own issue if not intended.

🤖 Generated with Claude Code

Lingua principale
C++
Stelle
241
Fork
62
Merge medio
1g 1h
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di charmplusplus/charm

Tutte le issue di charmplusplus/charm

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.