Distributed mode overwrites the user's choice for dataloaders shuffling
#10,170 opened on 2021幎10æ27æ¥
Repository metrics
- Stars
- Â (26,687 stars)
- PR merge metrics
-  (å¹³åããŒãž 9d 15h) (30d ã§ 3 merged PRs)
説æ
ð Bug
I ordered my training data in a specific manner and passed it to the DataLoader with shuffle=False (I use reload_dataloaders_every_n_epochs=1 to control it every epoch). Then, I found out that running in distributed mode re-creates the train dataloader and always set shuffle=True. The opposite happens for the val dataloader.
I think that the re-creation of the dataloaders must be using the same shuffle setting the original dataloaders had. Is there any reason not to do so?
Here is the related code with the hard-coded shuffle values:
train - shuffle forced to be true https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/trainer/data_loading.py#L330-L333
val - shuffle forced to be true https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/trainer/data_loading.py#L442-L443
cc @borda @justusschock @awaelchli