Lightning-AI/pytorch-lightning
Auf GitHub ansehenReturning None from training_step with multi GPU DDP training
Open
#5.243 geöffnet am 23. Dez. 2020
distributedfeaturehelp wantedpriority: 1
Repository-Metriken
- Stars
- (26.687 Stars)
- PR-Merge-Metriken
- (Durchschn. Merge 9T 15h) (3 gemergte PRs in 30 T)
Beschreibung
🐛 Bug
Returning None from training_step with multi GPU DDP training freezes the training without exception
To Reproduce
Starting multi-gpu training with a None-returning training_step function.
Example training_step function:
def training_step(self, batch, batch_idx):
data, target = batch
model_outputs = self.forward(images)
loss = calc_loss(model_outputs, target)
if torch.isnan(loss) or random.random() < .05:
return None
return loss
Example trainer:
trainer = Trainer(
gpus=2,
distributed_backend="ddp"
)
Expected behavior
To continue training with skipping the current batch as pointed out at here.
Environment
No specific environment is needed to reproduce this bug.
Additional context
This issue was mentioned here: #4956 but not with specifics.
Note: By the time this issue being investigated, a help for a workaround would be great!