Why Lingvo is so slow when training Librispeech960Wpm on a host with 6 GPUs

Aperta
  • Qualcosa non va?
#257 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python, tensorflow
Ambito
machine-learning

Direzione di ricerca

Start by reviewing the Librispeech960Wpm model configuration and the Lingvo trainer settings shown in the issue, especially the GPU, replica, and async options. Compare the reported steps per second with the data input and GPU utilization to identify where training time is spent. Done means providing a supported configuration or identifying a concrete bottleneck that explains the slow training.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

my host is one host, 6 GPUs(V100), the speed is about steps/sec: 0.12, so the 800,000 steps would take several months.

My running command is as follows:
bazel-bin/lingvo/trainer --saver_max_to_keep=3 --worker_gpus=6 --worker_replicas=3 --run_locally=gpu --mode=async --model=asr.librispeech.Librispeech960Wpm --logdir=/tmp/lingvo/asr_2 --logtostderr --enable_asserts=false --job=controller,trainer

through nvidia-smi: I can see that sin GPU "GPU-Util" are beteen 30%-60%

Any suggestions for the configuration?

Thanks

2021-03-19 00:03:38.408969: I lingvo/core/ops/record_yielder.cc:604] Record 1963: key=00001962
2021-03-19 00:03:39.040448: I lingvo/core/ops/record_yielder.cc:614] Emitted 2551 records from /tmp/lingvo/lingvo/speech_data/train/train.tfrecords-00034-of-00100
I0319 00:03:48.392294 139978034091776 summary_utils.py:398] Steps/second: 0.123965, Examples/second: 71.403573
I0319 00:03:48.394137 139978034091776 trainer_impl.py:199] step: 223, steps/sec: 0.12, examples/sec: 71.40 fraction_of_correct_next_step_preds:0.14076507 fraction_of_correct_next_step_preds/logits:0.14076507 grad_norm/all/loss:0.5464555 grad_scale_all/loss:1 has_nan_or_inf/loss:0 l2_loss/loss:0.58279711 learning_rate/loss:0.00025000001 log_pplx:6.1487246 log_pplx/logits:6.1487246 loss:6.1487246 loss/logits:6.1487246 num_samples_in_batch:576 token_normed_prob:0.0021416517 token_normed_prob/logits:0.0021416517 var_norm/all/loss:1079.627
2021-03-19 00:03:48.516815: I lingvo/core/ops/record_yielder.cc:604] Record 2094: key=00002093
I0319 00:03:57.226976 139978034091776 summary_utils.py:398] Steps/second: 0.124019, Examples/second: 71.434948
I0319 00:03:57.228809 139978034091776 trainer_impl.py:199] step: 224, steps/sec: 0.12, examples/sec: 71.43 fraction_of_correct_next_step_preds:0.14581797 fraction_of_correct_next_step_preds/logits:0.14581797 grad_norm/all/loss:0.53552681 grad_scale_all/loss:1 has_nan_or_inf/loss:0 l2_loss/loss:0.58274323 learning_rate/loss:0.00025000001 log_pplx:6.1034293 log_pplx/logits:6.1034293 loss:6.1034293 loss/logits:6.1034293 num_samples_in_batch:576 token_normed_prob:0.0022410566 token_normed_prob/logits:0.0022410566 var_norm/all/loss:1079.577

Lingua principale
Python
Stelle
2.9k
Fork
452
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tensorflow/lingvo

Tutte le issue di tensorflow/lingvo

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.