Colocated full-parameter weight sync: CPU fallback costs ~17% of every training step on seccomp-managed containers; request a pidfd_getfd-free GPU transport

Start with roll/utils/send_recv_utils.py:317-321 and roll/third_party/fsdp2/model_update.py:229, then inspect the existing _setup_broadcast_group infrastructure. Compare a non-pidfd_getfd transport for fully colocated full-parameter sync and benchmark it on the documented 3×4090 setup; done means avoiding mandatory CPU staging without breaking weight updates in the affected container environment.

Summary: On containers where pidfd_getfd is blocked (e.g. AutoDL managed instances, seccomp filter mode), colocated weight sync must use the ROLL_WEIGHT_SYNC_USE_CPU CPU-staging path. For full-parameter models this costs a measured ~103 s per step (~17% of the step wall time). This is a feature request for a GPU-side transport that does not depend on pidfd_getfd (e.g. NCCL broadcast or pinned-memory staging). Related: #484 reports the same class of problem on the Megatron colocated LoRA path with a different root cause (CUDA driver error on an accelerator environment).

Environment:

• ROLL @ 370cb24c1036ea9145365478fcc40612b2186fc8
• Actor: Qwen2.5-3B-Instruct, full-parameter PPO (agentic GRPO), bf16 weights ≈ 6.2 GB per sync
• FSDP2 actor training + vLLM actor inference, colocated (train ⊕ infer on GPU 0/1; fixed-role services on GPU 2), 3× RTX 4090
• vLLM 0.10.2, torch 2.8.0+cu128, ray 2.48.0
• AutoDL managed container, Seccomp: 2 (filter mode) — pidfd_getfd unavailable
• ROLL_WEIGHT_SYNC_USE_CPU: "1" (inherited through the example config chain)

Why the CPU path is mandatory here: roll/utils/send_recv_utils.py:317-319 already documents this:

# Managed AutoDL containers block pidfd_getfd, which makes CUDA IPC
# deserialization fail even for colocated workers. Preserve the default zero-copy
# path unless this explicit portable transport is requested.

The opt-in CPU path (send_recv_utils.py:320-321, bucket.detach().to("cpu").contiguous().numpy() + MultiprocessingSerializer, called from roll/third_party/fsdp2/model_update.py:229) works correctly — this is a performance request, not a bug report.

Measured cost (Qwen2.5-3B full-parameter, n=26 training steps):

→ weight sync is ~17% of every training step, and the single largest fixed per-step cost after rollout and the train pass. For LoRA-sized payloads this overhead is negligible (see #484: 169 MB), but it scales linearly with parameter count — at 7B+ full-parameter this would dominate.

Request: a colocated weight-sync transport that does not depend on pidfd_getfd, e.g.:

1. NCCL broadcast by extending the existing _setup_broadcast_group infrastructure in roll/third_party/fsdp2/model_update.py (currently separated/partial-overlap only) to fully-colocated mode, or
2. Pinned-memory staging with explicit cudaMemcpy (no IPC handles), or
3. Documented guidance on container runtimes that permit the syscalls CUDA IPC needs.

Happy to share the per-step metric dumps, the full step-time breakdown, or help benchmark alternatives — I have a reproducible 3×4090 colocated setup on the affected environment.