Integrating pytorch XLA when using multiple GPUs
#16,130 建立於 2022年12月20日
倉庫指標
- 星標
- (26,687 顆星)
- PR 合併指標
- (平均合併 9天 15小時) (30 天內合併 3 個 PR)
描述
Description & Motivation
I've experienced with pytorch XLA using multitple NVIDIA A100 GPU and I observed that in most cases training is faster. So it would be really nice to have the option to use XLA for training in pytorch lightning.
The main advantage is faster training.
Additional context
Here is a code link : https://github.com/Dhouib-med/Test-XLA/blob/17e5b6bd6c77fffa67818462856277a57877ff3b/test_xla.py to train a simple CNN on the MNIST dataset using XLA (on 2 GPUS). The main parts where taken from https://github.com/pytorch/xla. This wheel needs to be installed along with adequate pytorch and torchvision versions (1.11 and 0.14) https://storage.googleapis.com/tpu-pytorch/wheels/cuda/112/torch_xla-1.13-cp37-cp37m-linux_x86_64.whl @justusschock
cc @borda @justusschock @awaelchli @carmocca @JackCaoG @steventk-g @Liyang90