kaldi-asr/kaldi

Multi-GPU training capability for the Pytorch Transformer LM training script - https://github.com/kaldi-asr/kaldi/blob/master/egs/wsj/s5/local/pytorchnn/run_nnlm.sh

Open

#4,699 opened on Feb 16, 2022

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Shell (15,392 stars) (5,359 forks)batch import
enhancementhelp wantedstale-exclude

Description

I used the script ### https://github.com/kaldi-asr/kaldi/blob/master/egs/wsj/s5/local/pytorchnn/run_nnlm.sh, but I could not figure out how we could distribute the training of Transformer based LM on multiple GPUs in order to speed-up the Pytorch training. Please suggest if there is any way to do so.

Thanks!

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
unclear
Prerequisites
PyTorch basicsDistributed training concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Investigate adding multi GPU support to the PyTorch Transformer LM training script at kaldi/egs/wsj/s5/local/pytorchnn/run nnlm.sh. Review PyTorch's DistributedDataParallel (DDP) and how it can be integrated. Check if the script already has any GPU options. Look at comments from maintainers for guidance.
Multi-GPU training capability for the Pytorch Transformer LM training script - https://github.com/kaldi-asr/kaldi/blob/master/egs/wsj/s5/local/pytorchnn/run_nnlm.sh · kaldi-asr/kaldi#4699 | Good First Issue