Syncing the log_dir across ranks is not valid with multiple nodes

🐛 Bug

In TrainerProperties, we compute the log_dir, and then we overwrite it with rank 0's log_dir when returning the log_dir. This isn't valid in a multi-node setup because rank 0's log_dir may not exist on rank n's machine.

To Reproduce

We can use any model for this bug.

With 2 nodes

1. Create a unique temporary directory on each node as log_dir
2. Pass the unique log_dir into one of the lightning loggers and pass it into the trainer on each node
3. (possibly optional) pass a profiler into the trainer (to request log_dir and try to write to it)
4. Observe failure when we attempt to write something the log_dir on node 2

Expected behavior

If I pass a log_dir into the Logger or default_root_dir into the trainer, it should use that value.

Environment

• PyTorch Lightning Version (e.g., 1.3.0): 1.3.8 (still exists on master)
• PyTorch Version (e.g., 1.8) 1.6
• Python version: 3.6.9
• OS (e.g., Linux): Linux
• CUDA/cuDNN version: 10.1
• GPU models and configuration: Any model. 1 GPU per node, 2 nodes
• How you installed PyTorch (conda, pip, source): conda
• If compiling from source, the output of torch.__config__.show():
• Any other relevant information:

Additional context

https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1630348135165800