GPU helper threads inherit a PE's single-core affinity mask under +pemap, costing 22-63% of GPU iteration time

Start from HAPI first-device-call sites named in the issue (hapiInit, hapiCreateStreams, hapiEnqueue, hapiAddCallback, hapiPoolMalloc in src/arch/cuda/hybridAPI/hapi_impl.cpp) after CmiInitCPUAffinity. Helpers inherit the creating PE’s mask via pthread_create; the proposed fix is an opt-out widening scope using unpinned cores in the process cpuset (not SMT-only). CmiOnCore() is a stub in src/cpuaffinity.cpp, so PE-to-core mapping may need /proc/self/task/*. Done when GPU helpers are not pinned to a spinning PE under +pemap and a silent no-op is not used.

Summary

Under +pemap, GPU runtime helper threads are born pinned to a single worker PE's core and then timeslice against that PE for the life of the run. Measured cost on Frontier at 2B particles: −22% of iteration time at 16 nodes, and 2471.3 → 914.1 ms (−63%) at 128 nodes / 7168 PEs. An application-level workaround exists and is in production use, but the mechanism is a property of the runtime's affinity policy, not of any application, so it will recur for every GPU application that uses +pemap.

Mechanism

1. pthread_create gives a new thread its creator's CPU affinity mask (Linux).
2. GPU runtimes (ROCm/HIP confirmed; CUDA suspected) create their internal helper threads lazily, on the first real device call, from whichever thread happens to be making it — HSA queue servicing, signal/interrupt handlers, and similar.
3. Under Charm's +pemap that caller is a worker PE pinned to exactly one core, so each helper is born welded to that core.
4. A Charm PE with nothing to do spins rather than sleeps, so the PE and the helper then alternate at the scheduler timeslice (~10-16 ms).

The victim PE accumulated 2.7 seconds of runqueue wait in a 26-second run, while every PE with its core to itself accumulated 0.0 ms (/proc/<tid>/schedstat). Because collectives must reach the victim, the cost is roughly (collectives per run) x (one timeslice).

Why this is worse than the headline number suggests

It is a fixed-size tax per collective, so its share grows as phases shrink: in the same runs a ~5 ms gather phase inflated 21x while the treewalk inflated only 1.5x. It therefore hurts most exactly where optimization work is trying to make progress.

Second, independent benefit of fixing it: the fixed configuration was 7-10x more reproducible run-to-run, consistent with removing a stochastic preemption. Unfixed runs have variance large enough to swamp the A/B tests one wants to run.

Things that are NOT the cause (please don't re-chase)

• The GPU completion mechanism is not the gate. HAPI has two paths behind hapiAddCallback() (src/arch/cuda/hybridAPI/hapi_impl.cpp): with HAPI_CUDA_CALLBACK defined it calls hapiLaunchHostFunc, serviced by a driver-spawned callback thread; undefined, it calls recordEvent() and polls via hapiPollEvents on CcdSCHEDLOOP. The macro is defined nowhere in the build system, so the affected runs were already on the polling path and had the problem anyway — the runtime's internal threads are created on first device call regardless of how completions are delivered. Enabling callbacks can only add a thread class, never remove one.
• It is not a communication problem. It first presented as 53-81 ms machine-wide stalls at phase boundaries with no entry method executing anywhere, which looked like a network issue for a long time.

Application-level workaround (in production, and its flaw)

paratreet2 carries FoFHelperAffinityScope (fof/gpu/FoFDevice.cpp): an RAII scope around the first device work that widens the calling thread's cpuset so spawned helpers inherit the wide mask, then restores the PE's own mask on exit. Zone = SMT siblings of the pemap zone, minus any core holding a pinned PE, minus the caller's own core.

Two lessons from it that a runtime implementation should not repeat:

1. Do not gate on CPU_COUNT == 1. That version silently does nothing when a PE's mask has more than one CPU — which is exactly what Slurm's --cpu-bind=cores produces in multi-process runs, so the fix is inert in a common configuration. The right trigger is "the helper's inheritable mask overlaps cores that carry PEs".
2. Never decline silently. A no-op path that prints nothing is how a fix stops being applied without anyone noticing.

Proposed runtime fix

An opt-out affinity-widening scope in HAPI, latching per thread, at hapiInit plus the first-real-work sites (hapiCreateStreams, hapiEnqueue, hapiAddCallback, hapiPoolMalloc), and exported so applications that drive the GPU API directly (NAMD, for example) can wrap their own first device call. hapiInit runs after CmiInitCPUAffinity, so the pinning map is final by then.

Zone derivation must not assume SMT: prefer unpinned cores in the process cpuset, falling back to SMT siblings of PE cores where those exist, and declining with a warning when neither exists. The SMT-siblings-only rule used by the application version never fires on machines without SMT (e.g. Grace).

Known blocker for a clean implementation: CmiOnCore() is an unconditional return -1 stub in reconverse (src/cpuaffinity.cpp:784), so the PE-to-core map has to be recovered by reading /proc/self/task/* masks. Filed separately.

Platform / evidence

Frontier (AMD MI250X, ROCm), paratreet2 FoF at 2B particles, 16-128 nodes, 2026-08. Diagnosis chain: idle-exit binning -> an off-core detector using getrusage(RUSAGE_THREAD).ru_nivcsw -> /proc/<tid>/schedstat -> an LD_PRELOAD pthread_create interposer that logged each thread's creator and inherited mask.

A re-test on the current reviewed-with-reconverse + latest reconverse stack is in progress; the numbers above are from reconverse-specific-build. A CUDA-side case (NAMD on Grace-Hopper) is under diagnosis and unconfirmed.