Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[BUG]Loss connection to ranks

Open
  • Is something wrong?
#7,991 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

@Shreyas-S-809 is already working on this.

Since May 6, 2026.

  • #8127 by @lxcxjxhx — closed without merging

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python, pytorch
Domain
distributed-systems, machine-learning

Research direction

Start by reproducing the Transformers Trainer run with 16 A100s across two nodes, then inspect accelerate.utils.deepspeed at line 270 and deepspeed.runtime.stage_1_and_2.py around step() and scaled_global_norm(). Compare dist.get_rank() with the length of bit16_groups across ranks, including the one-node and four-node cases. Done means identifying why two ranks or parameter groups disappear and documenting a reliable fix or diagnostic.

Written by the indexing model from the issue text.

Description

bug training

Describe the bug
When I'm training my model with 16 gpu(2node), the backward will be stucked in

accelerate.utils.deepspeed line 270
self.engine.step()

So i step in and find out that in
deepspeed.runtime.stage_1_and_2.py line 2098

def step(self, closure=None):
        """
        Not supporting closure.
        """
        self.micro_step_id = INITIAL_MICRO_STEP_ID

        see_memory_usage("In step before checking overflow")

        # First compute norm for all group so we know if there is overflow
        if self.check_grad_overflow:
            self.check_overflow(partition_gradients=self.partition_gradients)

        prev_scale = self.loss_scale
        self._update_scale(self.overflow)
        if self.overflow:
            see_memory_usage('After overflow before clearing gradients')
            self.zero_grad(set_to_none=True)
            if self.cpu_offload:
                self.reset_cpu_buffers()
            else:
                for k in self.averaged_gradients.keys():
                    self.averaged_gradients[k] = None
                    self.all_grad_tensors[k] = None

            see_memory_usage('After overflow after clearing gradients')

            for timer in OPTIMIZER_TIMERS:
                self.timers(timer).start()
                self.timers(timer).stop()
            return

        # Step 1:- Calculate gradient norm using bit-16 grads
        see_memory_usage('Before norm calculation')
        scaled_global_grad_norm = self.scaled_global_norm()
        self._global_grad_norm = scaled_global_grad_norm / prev_scale
        see_memory_usage('After norm before optimizer')

In step 1, scaled_global_grad_norm = self.scaled_global_norm()

def scaled_global_norm(self, norm_type=2):
        assert norm_type == 2, "only L2 norm supported"
        norm_groups = []
        for i, group in enumerate(self.bit16_groups):
            if self.cpu_offload:
                norm = self.complete_grad_norm_calculation_for_cpu_offload(self.params_in_partition[i])
                norm_groups.append(norm)
            else:
                norm_groups.append(self.get_grad_norm_direct(self.averaged_gradients[i], self.params_in_partition[i]))

        if self.has_moe_layers:
            self._average_expert_grad_norms(norm_groups)

        # calculating L2 norm
        return torch.linalg.vector_norm(torch.stack(norm_groups), ord=norm_type)

I use pdb here, output dist.get_rank(), i can get all rank numbers(0~15), however, if i use command
pirnt(self.bit16_groups), it will only give 14 results, and I will loss connections. I don't know what triggered this.

To Reproduce

  1. add SAM3 to any MLLMs, just give the last hidden_states to SAM3 video predictor(use projector)
  2. start training with 2nodes(16 gpus)
  3. Note: I can't reproduce this in 1 node or 4 nodes

Expected behavior
stuck at the 9th step

ds_report output

Screenshots

System info (please complete the following information):

  • GPU count and types [16 A100s(2 nodes)]
  • Python version 3.10.4

Launcher context
using transformers trainer

Dominant language
Python
Stars
43.3k
Forks
5.1k
Avg merge
4d 20h
Merged PRs (30d)
69

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepspeedai/DeepSpeed

All issues in deepspeedai/DeepSpeed

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.