[bug]Assert error run test_internode.py on EP128

Open Beginner friendly
#518 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
62/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
distributed-systems

Research direction

Start with test_internode.py and locate the nvl_buffer_size selection and the nvl_chunk_size tuning loop. Run the normal-kernel EP128 case to reproduce the assertion, then verify the buffer-size adjustment prevents the failure without breaking the other listed rank configurations.

Written by the indexing model from the issue text.

Description

when run normal kernel use EP128, this assert will fail

Assertion error  num_max_nvl_chunked_recv_tokens / num_rdma_ranks > std::max(num_max_rdma_chunked_send_tokens, num_max_nvl_chunked_send_tokens)'

because the num_max_nvl_chunked_send_tokens will be set to bigger than num_max_nvl_chunked_recv_tokens / num_rdma_ranks when tuning

for nvl_chunk_size in range(4, *45*, 4):  // test_internode.py

it seems should add nvl_buffer_size to fix it

rdma_buffer_size, nvl_buffer_size = 128, (720 if num_ranks in (24, 48, 96, 144, 160) else 512) 
-> rdma_buffer_size, nvl_buffer_size = 128, (720 if num_ranks in (24, 48, 96, 128,144, 160) else 512)
Dominant language
Cuda
Stars
10.2k
Forks
1.4k
Avg merge
2d 17h
Merged PRs (30d)
3

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepseek-ai/DeepEP

All issues in deepseek-ai/DeepEP

Similar issues

More Distributed Systems issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.