Lightning-AI/pytorch-lightning

Leak RAM when training 8 core TPU

Open

#16 876 ouverte le 26 févr. 2023

Voir sur GitHub
 (3 commentaires) (0 réactions) (0 assignés)Python (3 233 forks)batch import
accelerator: tpubughelp wantedperformancerepro needed

Métriques du dépôt

Stars
 (26 687 stars)
Métriques de merge PR
 (Merge moyen 9j 15h) (3 PRs mergées en 30 j)

Description

Bug description

I am training on 8 TPU cores but the memory CPU Leak when the epoch ends.

Model

mobilenetV2

Data

cifar100

Env

pytorch_lightning==1.9.3 pytorch==1.13.0 code run on google colab

How to reproduce the bug

No response

Error messages and logs

# Error messages and logs here please

Environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info

No response

cc @carmocca @JackCaoG @Liyang90 @gkroiz @borda @steventk-g

Guide contributeur