Lightning-AI/pytorch-lightning
Auf GitHub ansehenSyncing the log_dir across ranks is not valid with multiple nodes
Open
#9.212 geöffnet am 30. Aug. 2021
bugdistributedhelp wantedloggingpriority: 1
Repository-Metriken
- Stars
- (26.687 Stars)
- PR-Merge-Metriken
- (Durchschn. Merge 9T 15h) (3 gemergte PRs in 30 T)
Beschreibung
🐛 Bug
In TrainerProperties, we compute the log_dir, and then we overwrite it with rank 0's log_dir when returning the log_dir. This isn't valid in a multi-node setup because rank 0's log_dir may not exist on rank n's machine.
To Reproduce
We can use any model for this bug.
With 2 nodes
- Create a unique temporary directory on each node as log_dir
- Pass the unique log_dir into one of the lightning loggers and pass it into the trainer on each node
- (possibly optional) pass a profiler into the trainer (to request log_dir and try to write to it)
- Observe failure when we attempt to write something the log_dir on node 2
Expected behavior
If I pass a log_dir into the Logger or default_root_dir into the trainer, it should use that value.
Environment
- PyTorch Lightning Version (e.g., 1.3.0): 1.3.8 (still exists on master)
- PyTorch Version (e.g., 1.8) 1.6
- Python version: 3.6.9
- OS (e.g., Linux): Linux
- CUDA/cuDNN version: 10.1
- GPU models and configuration: Any model. 1 GPU per node, 2 nodes
- How you installed PyTorch (
conda,pip, source): conda - If compiling from source, the output of
torch.__config__.show(): - Any other relevant information:
Additional context
https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1630348135165800