Incorrect bus bandwidth calculation for subgroup collectives
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 4/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Domain
- distributed-systems
Research direction
Read deepspeed/comm/comm.py::timed_op() and deepspeed/utils/comms_logging.py::calc_bw_log() to trace how the process group reaches the bandwidth calculation. The reproduction in the issue gives a 4-process world and 2-process subgroup; use it to check the expected bus bandwidth of 0.004096 Gbps and verify summary records remain separate by subgroup size. A linked pull request is already open, so the fix is under way.
Written by the indexing model from the issue text.
Description
Problem
DeepSpeed's communication logger can report incorrect bus bandwidth for collectives that run on a subgroup.
deepspeed/utils/comms_logging.py::calc_bw_log() currently calculates bandwidth using dist.get_world_size() without the collective's process group. When a collective operates on a subgroup, the calculation therefore uses the global world size instead of the number of participating processes.
The process group is available in deepspeed/comm/comm.py::timed_op(), but it was not being propagated to the communication logger. In a focused reproduction with a global world size of 4 and a subgroup size of 2, an all_reduce reported a bus bandwidth of 0.006144 Gbps instead of the expected 0.004096 Gbps.
Expected behavior
Bandwidth calculations should use the size of the process group participating in the collective.
The change should also preserve existing behavior for world-group collectives and callers that omit the group. Summary records for different subgroup sizes should not be combined, since they use different bandwidth formulas.
Contributor context
I am opening this issue primarily to give maintainers context about the bug and its root cause. I have already implemented a proposed fix locally and intend to submit it as a separate pull request referencing this issue.
To avoid duplicated effort, contributors are kindly asked to refrain from independently implementing a fix for this specific issue while that pull request is being prepared and reviewed. Maintainer feedback and discussion about the appropriate solution are very welcome.
Scope
This issue is focused on incorrect subgroup bandwidth calculations and summary separation by subgroup size.
Distinguishing separate process groups that have the same size, and subgroup-aware straggler analysis, are outside the scope of the proposed fix.
Minimal reproduction
The issue can be reproduced with a 4-process distributed setup where the collective uses a subgroup of 2 processes. The following focused calculation models that situation against the current implementation:
import deepspeed.comm as dist
from deepspeed.utils.comms_logging import calc_bw_log
# The current implementation queries the global world size.
dist.get_world_size = lambda: 4
size = 1024 # bytes
duration = 2.0 # seconds
print(calc_bw_log("all_reduce", size, duration))
Current output:
(0.008192, 0.006144)
The algorithmic bandwidth is unchanged, but for a subgroup of size 2 the expected bus bandwidth is:
0.004096 Gbps
The discrepancy comes from the all-reduce bus-bandwidth formula using n=4 instead of the subgroup size n=2. A distributed reproduction should create a 4-process job, construct a 2-process process group, and run an all_reduce through the communication logger on that group.
- Dominant language
- Python
- Stars
- 43.3k
- Forks
- 5.1k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 63
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeed
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
deepspeedai/DeepSpeed#8925 · 1 comment ·
Maintainers usually reply within 3 days
-
[BUG] Universal checkpoint resume puts CPU `step` into fused/capturable optimizer statePossibly taken @ArjunPakhan claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
deepspeedai/DeepSpeed#8884 · 1 comment ·
Maintainers usually reply within 3 days
-
init_inference rejects dtype=torch.int8 on every acceleratorMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8577 ·
Maintainers usually reply within 3 days
-
[BUG] AutoTP checkpoint load raises AttributeError on a model containing nn.InstanceNorm1dPossibly taken @vineethsaivs claimed this 45 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
deepspeedai/DeepSpeed#8514 ·
Maintainers usually reply within 3 days
-
[BUG] Python checkpoint writers fail to reload full ZeRO-3 state with default torch.loadPossibly taken @ai-yang claimed this 46 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8500 ·
Maintainers usually reply within 3 days
All issues in deepspeedai/DeepSpeed
Similar issues
-
enhancement help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Git-Hospital/githealthcheck#61 ·
Maintainers usually reply within 1 day
-
changelog investigate
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
ramnes/notion-sdk-py#433 ·
-
Partial method descriptors raise on class accessPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 19 days
-
docs pkg:core
Difficulty 1/5 Under an hour Newbie friendliness 84/100
pydantic/pydantic-ai#11929 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
intake/intake-esm#789 ·