Lightning-AI/pytorch-lightning

BUG when Trainer.test() with deepspeed stage 3

Open

#14,090 opened on Aug 8, 2022

View on GitHub
 (2 comments) (2 reactions) (0 assignees)Python (3,233 forks)batch import
bughelp wantedrepro neededstrategy: deepspeed

Repository metrics

Stars
 (26,687 stars)
PR merge metrics
 (Avg merge 9d 15h) (3 merged PRs in 30d)

Description

🐛 Bug

Hi, I meet some bugs when I combine lightning with deepspeed follow https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#trainer-class-api.

I find that:

  • Trainer.fit() is compatible well with deepspeed stage 2 and 3 .
  • Trainer.test() is compatible with deepspeed stage 2.
  • There is some bugs when combine Trainer.test() with deepspeed stage 3:

When initialize DeepSpeedStrategy with deepspeed config and then test, I will encounter this bug, I have tried to change the config, but this bug is exsited whatever.

image

When initialize DeepSpeedStrategy with lightning args, I will encounter two problems, the generate speed is too slow (I used huggingface transformers and their generate() method) than the deepspeed inference scripts in https://www.deepspeed.ai/tutorials/inference-tutorial/. In addition, I also meet a bug called index of range because of self.__step of deepspeed engine.

Hope for better compatibility between lightning and deepspeed !

cc @awaelchli @rohitgr7 @akihironitta

Contributor guide

Research direction
Investigate the compatibility of Trainer.test() with DeepSpeed stage 3 by examining the DeepSpeed integration in Lightning's testing loop. Check for differences compared to stage 2 and the fit method. Look at the DeepSpeed engine's step counter and any potential issues with gradient accumulation or model state during evaluation.
Tech stack
pythonpytorch
Domain
backendinfrastructure
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonPyTorchDeepSpeedPyTorch Lightning
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60