Lightning-AI/pytorch-lightning

`batch_sampler.batch_size` is None with deepspeed and `DataLoader(batch_size=None)`

Open

#19.460 aberto em 13 de fev. de 2024

Ver no GitHub
 (4 comments) (0 reactions) (0 assignees)Python (3.233 forks)batch import
bughelp wantedstrategy: deepspeed

Métricas do repositório

Stars
 (26.687 stars)
Métricas de merge de PR
 (Mesclagem média 9d 15h) (3 fundiu PRs em 30d)

Description

Bug description

Hello,

After Lightning 2.2.0 upgrade we experience a crash when using deepspeed with DataLoader(batch_size=None, ...):

  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run
    self.strategy.setup(self)
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 335, in setup
    self._init_config_if_needed()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 804, in _init_config_if_needed
    self._format_config()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 813, in _format_config
    self._format_batch_size_and_grad_accum_config()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 906, in _format_batch_size_and_grad_accum_config
    batch_size = self._auto_select_batch_size()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 920, in _auto_select_batch_size
    batch_size = train_dataloader.batch_sampler.batch_size
AttributeError: 'NoneType' object has no attribute 'batch_size'

This was working on 2.1 versions

What version are you seeing the problem on?

v2.2

How to reproduce the bug

class DataModule(LightningDataModule):
    def train_dataloader(self) -> DataLoader:
        # Don't set batch size here, it's done in the datapipe
        return DataLoader(
            self.train_dp, batch_size=None,...
        )

trainer = Trainer(strategy='deepspeed', ...)
trainer.fit(model, DataModule())

Error messages and logs

  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run
    self.strategy.setup(self)
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 335, in setup
    self._init_config_if_needed()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 804, in _init_config_if_needed
    self._format_config()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 813, in _format_config
    self._format_batch_size_and_grad_accum_config()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 906, in _format_batch_size_and_grad_accum_config
    batch_size = self._auto_select_batch_size()
  File ".../pip-ai-experimental_pytorch_lightning/site-packages/pytorch_lightning/strategies/deepspeed.py", line 920, in _auto_select_batch_size
    batch_size = train_dataloader.batch_sampler.batch_size
AttributeError: 'NoneType' object has no attribute 'batch_size'

Environment

Unfortunately, I can't collect the environment because we use a custom build system :(

lightning==2.2.0 torch==2.1.2

More info

No response

cc @awaelchli

Guia do colaborador

Direção de pesquisa
Investigue o método ` auto select batch size` em `deepspeed.py`. Quando `batch sampler.batch size` é `None`, o método deve recorrer à obtenção do tamanho do lote a partir do atributo `batch size` do dataloader ou do conjunto de dados. Verifique a lógica em ` format batch size and grad accum config` para lidar com esse caso adequadamente.
Pilha de tecnologia
python
Domain
backend
Tipo Issue
Bug
DifficultyDificuldade de implementação estimada para um novo contribuidor, de 1 para alterações muito pequenas a 5 para trabalho de nível especializado.
2
Tempo estimadoUm intervalo de tempo aproximado para um colaborador experiente investigar, implementar, testar e preparar um pull request.
1-3 horas
Status da atividadeQuão disponível o issue aparece agora: novo, ativo, obsoleto, bloqueado ou aguardando entrada do mantenedor.
Ativo
ClarityCom que clareza o issue explica a mudança esperada, os critérios de aceitação e a próxima etapa.
Claro
Prerequisites
PythonPyTorch LightningDeepSpeed
Simpatia para novatosUma pontuação de 1 a 100 que estima o quão acessível este issue é para colaboradores iniciantes.
60