Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Incorrect bus bandwidth calculation for subgroup collectives

Aperta
  • Qualcosa non va?
#8,861 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@marcusacosta ci sta già lavorando.

Dal 19/10/2026.

  • #8862 di @marcusacosta — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
python, pytorch
Ambito
distributed-systems

Direzione di ricerca

Read deepspeed/comm/comm.py::timed_op() and deepspeed/utils/comms_logging.py::calc_bw_log() to trace how the process group reaches the bandwidth calculation. The reproduction in the issue gives a 4-process world and 2-process subgroup; use it to check the expected bus bandwidth of 0.004096 Gbps and verify summary records remain separate by subgroup size. A linked pull request is already open, so the fix is under way.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

DeepSpeed's communication logger can report incorrect bus bandwidth for collectives that run on a subgroup.

deepspeed/utils/comms_logging.py::calc_bw_log() currently calculates bandwidth using dist.get_world_size() without the collective's process group. When a collective operates on a subgroup, the calculation therefore uses the global world size instead of the number of participating processes.

The process group is available in deepspeed/comm/comm.py::timed_op(), but it was not being propagated to the communication logger. In a focused reproduction with a global world size of 4 and a subgroup size of 2, an all_reduce reported a bus bandwidth of 0.006144 Gbps instead of the expected 0.004096 Gbps.

Expected behavior

Bandwidth calculations should use the size of the process group participating in the collective.

The change should also preserve existing behavior for world-group collectives and callers that omit the group. Summary records for different subgroup sizes should not be combined, since they use different bandwidth formulas.

Contributor context

I am opening this issue primarily to give maintainers context about the bug and its root cause. I have already implemented a proposed fix locally and intend to submit it as a separate pull request referencing this issue.

To avoid duplicated effort, contributors are kindly asked to refrain from independently implementing a fix for this specific issue while that pull request is being prepared and reviewed. Maintainer feedback and discussion about the appropriate solution are very welcome.

Scope

This issue is focused on incorrect subgroup bandwidth calculations and summary separation by subgroup size.

Distinguishing separate process groups that have the same size, and subgroup-aware straggler analysis, are outside the scope of the proposed fix.

Minimal reproduction

The issue can be reproduced with a 4-process distributed setup where the collective uses a subgroup of 2 processes. The following focused calculation models that situation against the current implementation:

import deepspeed.comm as dist
from deepspeed.utils.comms_logging import calc_bw_log

# The current implementation queries the global world size.
dist.get_world_size = lambda: 4

size = 1024       # bytes
duration = 2.0    # seconds

print(calc_bw_log("all_reduce", size, duration))

Current output:

(0.008192, 0.006144)

The algorithmic bandwidth is unchanged, but for a subgroup of size 2 the expected bus bandwidth is:

0.004096 Gbps

The discrepancy comes from the all-reduce bus-bandwidth formula using n=4 instead of the subgroup size n=2. A distributed reproduction should create a 4-process job, construct a 2-process process group, and run an all_reduce through the communication logger on that group.

Lingua principale
Python
Stelle
43.3k
Fork
5.1k
Merge medio
4g 4h
PR unite (30g)
65

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di deepspeedai/DeepSpeed

Tutte le issue di deepspeedai/DeepSpeed

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.