【bug】Engram wait can return before fetched data is visible (memory_order_relaxed in engram_fetch_wait.cuh)
Maintainers usually reply within 10 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- cpp
- Domain
- distributed-systems
Research direction
Read deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh at the gin.wait call and check how its memory order affects visibility of fetched data. Reproduce with the stress check described in the issue, then verify the reported acquire-order change against the default and forward wait-order cases. Done means fetched rows match the reference without synchronization or added delay.
Written by the indexing model from the issue text.
Description
Engram wait can return before fetched data is visible (memory_order_relaxed in engram_fetch_wait.cuh)
Summary
With DeepEP v2.5.0 on an H800 + RoCE cluster, EngramBuffer.fetch() wait hooks can return while the receive
buffer still holds stale/poison contents. Changing one line in
deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh from cuda::memory_order_relaxed to
cuda::memory_order_acquire eliminates the failure (8/8 GPUs, 400 iterations each, no measurable overhead).
Adding a system-scope acquire fence after the wait loop has the same effect.
Environment
- DeepEP v2.5.0, commit
f6aa38f, built from source (wheel,bash install.sh) - NVIDIA H800, 8 GPUs, single process / single rank per GPU (
world_size=1) - 8x 400G ConnectX-7
- CUDA 13.1, PyTorch 2.10.0a0, NCCL 2.32.6 (compile-time and runtime identical)
Reproduction
Stress check (deepep_engram_check.py-style):
EngramBuffer(group, gpu_bytes, host_bytes, use_cpu_rdma_storage=True, num_allocated_qps=32, qp_depth=8192)
withlayers=2,entries=4,rows=2^20,hidden=288(576 bytes/row).buffer.write(tables)with random uint8 data.- 400 iterations: random
indices [layers, tokens, entries](token count sweeps boundaries and random values
up tomax_t=8192), poison the returned receive view with0xA5, then
hooks = buffer.fetch(indices, num_qps=32),data = hooks[layer]().view(uint8)andclone()it,
and compare against anindex_selectreference. - Count rows with any mismatch.
All work is stream-ordered on the current stream; there is no CPU sync or other work between hook() and
clone().
Result
Failure on 8/8 GPUs, per-GPU:
bad_iters 162 – 178 / 400
bad_rows 892 – 1362
late_bad_rows 0 (a second read after the check is always correct)
all_poison True for most bad rows
late_bad_rows=0 means the data does arrive later; it is simply not visible when wait returns.
all_poison=True means the whole row still contains the poison pattern.
Controls
| Variant | bad_iters / 400 |
|---|---|
default (--wait-order=alternate) |
169 |
--wait-order=forward |
332 |
--wait-order=reverse |
0 |
--qps=8 |
0 |
--qps=64 |
149 |
--tokens=512 |
0 |
--tokens=1024 |
198 |
use_cpu_rdma_storage=False (GPU-resident storage) |
184 |
background matmul on a side stream (--load=1) |
0 (3 runs) |
NCCL_GIN_TYPE=3 (GDAKI backend instead of the CPU proxy) |
170 |
--mode=no-wait (negative control) |
378 (expected) |
torch.cuda.synchronize() between hook() and clone() |
0 |
time.sleep(0.05) between hook() and clone() |
0 |
Two observations:
- The failure is not specific to CPU/host storage, nor to the CPU proxy backend (GDAKI fails the same way).
- Any added host-side delay hides it, which points to a completion-signal vs data-visibility window rather
than an addressing problem.
Root cause
deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh:26:
gin.wait(last_gin_req, cuda::memory_order_relaxed);
The relaxed poll provides atomicity but no acquire ordering. Once the completion counter satisfies the wait,
subsequent reads of the receive buffer are not ordered after the payload becoming visible; on this platform
the reads can observe the pre-DMA contents (the poison values, which are still resident in L2) even though
the completion counter has advanced.
The GIN wait API already takes a cuda::memory_order, and in the proxy backend it forwards it to
ci.load(ord); the GDAKI backend applies cuda::atomic_thread_fence(ord, thread_scope_system) on success,
which is a no-op with relaxed. So nothing below NCCL changes between relaxed and acquire; the
difference is entirely in the ordering the caller requests.
It is plausible that this is a latent issue on all platforms and only becomes observable when the completion
signal can be seen before the payload is committed (PCIe relaxed ordering, IOMMU/virtualized root complex,
L2 residency of the destination lines). That would explain why it is intermittent and parameter-sensitive
here and does not reproduce elsewhere.
Proposed fix
--- a/deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh
+++ b/deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh
@@ -23,7 +23,7 @@ engram_fetch_wait_impl(const ncclDevComm_t nccl_dev_comm,
auto last_gin_req_int4 = __ldg(reinterpret_cast<int4*>(last_gin_requests + peer_context_idx));
auto last_gin_req = *reinterpret_cast<ncclGinRequest_t*>(&last_gin_req_int4);
- gin.wait(last_gin_req, cuda::memory_order_relaxed);
+ gin.wait(last_gin_req, cuda::memory_order_acquire);
}
}
Verification of the fix
acquireonly: 0/400 on all 8 GPUs (seeds 0–7); runtime unchanged (0.76–0.82 s vs 0.79–0.85 s).- Also passes the previously worst cases:
--wait-order=forward,--qps=64,--tokens=1024. - Reverting to
relaxedimmediately reproduces 160/400 — the change is causal.
engram_fetch_wait.cuh is the only place under deep_ep/include that uses memory_order_relaxed; other
gin.wait call sites pass an explicit order.
Notes
- A full
cudaDeviceSynchronize()or a 50 ms sleep after the hook also makes the test pass, but these only
widen the window and are not acceptable on a latency-critical path. - The API contract of
fetch()/hook()is "after the hook, the buffer is readable", soacquireseems to
match the intended semantics.
- Dominant language
- Cuda
- Stars
- 10.5k
- Forks
- 1.6k
- Avg merge
- 12d 23h
- Merged PRs (30d)
- 3
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepseek-ai/DeepEP
-
`EP_BUFFER_DEBUG=0` enables Python-side debug output because the env var is tested as a raw stringPossibly taken @yashkgp claimed this 103 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
deepseek-ai/DeepEP#718 · 1 comment ·
Maintainers usually reply within 10 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
deepseek-ai/DeepEP#675 ·
Maintainers usually reply within 10 days
-
official dockerfilePossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-2 days Newbie friendliness 66/100
deepseek-ai/DeepEP#615 · 1 comment ·
Maintainers usually reply within 10 days
-
EPv2: get_rdma_gbs() get half bandwidth in LACP bonding environmentsPossibly taken @zfan2356 claimed this 148 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
deepseek-ai/DeepEP#614 ·
Maintainers usually reply within 10 days
-
[bug]Assert error run test_internode.py on EP128Possibly taken @yurekami claimed this 330 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
deepseek-ai/DeepEP#518 ·
Maintainers usually reply within 10 days
All issues in deepseek-ai/DeepEP
Similar issues
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
kubernetes-sigs/kueue#19255 ·
Maintainers usually reply within 1 day
-
Controller Deployment discovery rejects shorter pod-template hashesPossibly taken @c-kruse claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
skupperproject/skupper#2641 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
dpkp/kafka-python#3203 ·
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bghira/SimpleTuner#3345 ·
Maintainers usually reply within 1 day
-
Push can publish a ref whose objects were never uploaded if the local branch moves during the pushOpen
Difficulty 2/5 1-3 hours Newbie friendliness 76/100