Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[BUG] crashed training gpt model using 3d parallelization

Open
  • Is something wrong?
#7,922 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
python, pytorch
Domain
distributed-systems, machine-learning

Research direction

Start by collecting the requested ds_report, system information, launcher and Docker context, then create a minimal reproduction for the four-stage pipeline, four-way model parallelism, and ZeRO stage 1 setup. Inspect the reported deepspeed/runtime/engine.py _backward_post_hook path and the training call that invokes backward; done means the failure is reproduced with enough configuration detail to identify whether the setup or DeepSpeed behavior is responsible.

Written by the indexing model from the issue text.

Description

bug training

I always failed to use 3d (using pipe stages 4, mp 4, and zero stage 1) to pretrain gpt model, the following is the crashed message,

[rank54]: File "/.../miniconda3/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2392, in _backward_post_hook

[rank54]: raise RuntimeError(error_msg)

[rank54]: RuntimeError: Loss scaling is required for this configuration, but backward() was called directly without scaling the loss. Please use one of the following: 1. engine.backward(loss) 2. engine.scale(loss).backward()

[rank88]:[W325 16:06:00.516742289 ProcessGroupNCCL.cpp:3955] Warning: [PG ID 0 PG GUID 0(default_pg) Rank 88] An unbatched P2P op (send/recv) was called on this ProcessGroup with size 128. In eager initialization mode, unbatched P2P ops are treated as independent collective ops, and are thus serialized with all other ops on this ProcessGroup, including other P2P ops. To avoid serialization, either create additional independent ProcessGroups for the P2P ops or use batched P2P ops. You can squash this warning by setting the environment variable TORCH_NCCL_SHOW_EAGER_INIT_P2P_SERIALIZATION_WARNING to false. (function operator())

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

ds_report output
Please run ds_report to give us details about your setup.

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

  • OS: [e.g. Ubuntu 18.04]
  • GPU count and types [e.g. two machines with x8 A100s each]
  • Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
  • Python version
  • Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?

Docker context
Are you using a specific docker image that you can share?

Additional context
Add any other context about the problem here.

Dominant language
Python
Stars
43.3k
Forks
5.1k
Avg merge
4d 20h
Merged PRs (30d)
69

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepspeedai/DeepSpeed

All issues in deepspeedai/DeepSpeed

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.