Lightning-AI/pytorch-lightning

Distributed mode overwrites the user's choice for dataloaders shuffling

Open

#10,170 创建于 2021年10月27日

在 GitHub 查看
 (8 评论) (0 反应) (0 负责人)Python (3,233 fork)batch import
data handlingfeaturehelp wanted

仓库指标

Star
 (26,687 star)
PR 合并指标
 (平均合并 9天 15小时) (30 天内合并 3 个 PR)

描述

🐛 Bug

I ordered my training data in a specific manner and passed it to the DataLoader with shuffle=False (I use reload_dataloaders_every_n_epochs=1 to control it every epoch). Then, I found out that running in distributed mode re-creates the train dataloader and always set shuffle=True. The opposite happens for the val dataloader.

I think that the re-creation of the dataloaders must be using the same shuffle setting the original dataloaders had. Is there any reason not to do so?


Here is the related code with the hard-coded shuffle values:

train - shuffle forced to be true https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/trainer/data_loading.py#L330-L333

val - shuffle forced to be true https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/trainer/data_loading.py#L442-L443

cc @borda @justusschock @awaelchli

贡献者指南