【bug】Engram wait can return before fetched data is visible (memory_order_relaxed in engram_fetch_wait.cuh)

Open Beginner friendly
#830 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 10 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp

Research direction

Read deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh at the gin.wait call and check how its memory order affects visibility of fetched data. Reproduce with the stress check described in the issue, then verify the reported acquire-order change against the default and forward wait-order cases. Done means fetched rows match the reference without synchronization or added delay.

Written by the indexing model from the issue text.

Description

Engram wait can return before fetched data is visible (memory_order_relaxed in engram_fetch_wait.cuh)

Summary

With DeepEP v2.5.0 on an H800 + RoCE cluster, EngramBuffer.fetch() wait hooks can return while the receive
buffer still holds stale/poison contents. Changing one line in
deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh from cuda::memory_order_relaxed to
cuda::memory_order_acquire eliminates the failure (8/8 GPUs, 400 iterations each, no measurable overhead).
Adding a system-scope acquire fence after the wait loop has the same effect.

Environment

  • DeepEP v2.5.0, commit f6aa38f, built from source (wheel, bash install.sh)
  • NVIDIA H800, 8 GPUs, single process / single rank per GPU (world_size=1)
  • 8x 400G ConnectX-7
  • CUDA 13.1, PyTorch 2.10.0a0, NCCL 2.32.6 (compile-time and runtime identical)

Reproduction

Stress check (deepep_engram_check.py-style):

  1. EngramBuffer(group, gpu_bytes, host_bytes, use_cpu_rdma_storage=True, num_allocated_qps=32, qp_depth=8192)
    with layers=2, entries=4, rows=2^20, hidden=288 (576 bytes/row).
  2. buffer.write(tables) with random uint8 data.
  3. 400 iterations: random indices [layers, tokens, entries] (token count sweeps boundaries and random values
    up to max_t=8192), poison the returned receive view with 0xA5, then
    hooks = buffer.fetch(indices, num_qps=32), data = hooks[layer]().view(uint8) and clone() it,
    and compare against an index_select reference.
  4. Count rows with any mismatch.

All work is stream-ordered on the current stream; there is no CPU sync or other work between hook() and
clone().

Result

Failure on 8/8 GPUs, per-GPU:

bad_iters   162 – 178 / 400
bad_rows    892 – 1362
late_bad_rows 0            (a second read after the check is always correct)
all_poison  True            for most bad rows

late_bad_rows=0 means the data does arrive later; it is simply not visible when wait returns.
all_poison=True means the whole row still contains the poison pattern.

Controls

Variant bad_iters / 400
default (--wait-order=alternate) 169
--wait-order=forward 332
--wait-order=reverse 0
--qps=8 0
--qps=64 149
--tokens=512 0
--tokens=1024 198
use_cpu_rdma_storage=False (GPU-resident storage) 184
background matmul on a side stream (--load=1) 0 (3 runs)
NCCL_GIN_TYPE=3 (GDAKI backend instead of the CPU proxy) 170
--mode=no-wait (negative control) 378 (expected)
torch.cuda.synchronize() between hook() and clone() 0
time.sleep(0.05) between hook() and clone() 0

Two observations:

  • The failure is not specific to CPU/host storage, nor to the CPU proxy backend (GDAKI fails the same way).
  • Any added host-side delay hides it, which points to a completion-signal vs data-visibility window rather
    than an addressing problem.

Root cause

deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh:26:

gin.wait(last_gin_req, cuda::memory_order_relaxed);

The relaxed poll provides atomicity but no acquire ordering. Once the completion counter satisfies the wait,
subsequent reads of the receive buffer are not ordered after the payload becoming visible; on this platform
the reads can observe the pre-DMA contents (the poison values, which are still resident in L2) even though
the completion counter has advanced.

The GIN wait API already takes a cuda::memory_order, and in the proxy backend it forwards it to
ci.load(ord); the GDAKI backend applies cuda::atomic_thread_fence(ord, thread_scope_system) on success,
which is a no-op with relaxed. So nothing below NCCL changes between relaxed and acquire; the
difference is entirely in the ordering the caller requests.

It is plausible that this is a latent issue on all platforms and only becomes observable when the completion
signal can be seen before the payload is committed (PCIe relaxed ordering, IOMMU/virtualized root complex,
L2 residency of the destination lines). That would explain why it is intermittent and parameter-sensitive
here and does not reproduce elsewhere.

Proposed fix

--- a/deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh
+++ b/deep_ep/include/deep_ep/impls/engram/engram_fetch_wait.cuh
@@ -23,7 +23,7 @@ engram_fetch_wait_impl(const ncclDevComm_t nccl_dev_comm,
         auto last_gin_req_int4 = __ldg(reinterpret_cast<int4*>(last_gin_requests + peer_context_idx));
         auto last_gin_req = *reinterpret_cast<ncclGinRequest_t*>(&last_gin_req_int4);
-        gin.wait(last_gin_req, cuda::memory_order_relaxed);
+        gin.wait(last_gin_req, cuda::memory_order_acquire);
     }
 }
Verification of the fix
  • acquire only: 0/400 on all 8 GPUs (seeds 0–7); runtime unchanged (0.76–0.82 s vs 0.79–0.85 s).
  • Also passes the previously worst cases: --wait-order=forward, --qps=64, --tokens=1024.
  • Reverting to relaxed immediately reproduces 160/400 — the change is causal.

engram_fetch_wait.cuh is the only place under deep_ep/include that uses memory_order_relaxed; other
gin.wait call sites pass an explicit order.

Notes

  • A full cudaDeviceSynchronize() or a 50 ms sleep after the hook also makes the test pass, but these only
    widen the window and are not acceptable on a latency-critical path.
  • The API contract of fetch()/hook() is "after the hook, the buffer is readable", so acquire seems to
    match the intended semantics.
Dominant language
Cuda
Stars
10.5k
Forks
1.6k
Avg merge
12d 23h
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepseek-ai/DeepEP

All issues in deepseek-ai/DeepEP

Similar issues

More Distributed Systems issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.