Unable to run LM DenseLm128B8x8

Aperta
  • Qualcosa non va?
#267 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
28/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python, tensorflow
Ambito
distributed-systems, machine-learning

Direzione di ricerca

Start by tracing the checkpoint save path shown in the traceback, from lingvo/executor.py through lingvo/core/checkpointer.py, and compare it with the TensorFlow session failure in the log. Reproduce the run with the reported model and TPU settings if the setup is available; done means identifying why saving loses the session and confirming a reliable run or a documented workaround.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hi,

I'm interested in seeing the timing relationship between layers and operations of the lm denselm128b8x8 model.
I was using a v3-8 node and changed the hparams (num_device_per_split, mesh_shape) in Task() and also changed the num of training steps to save time.

However, after initializing the vars, I got this error (here is the log):

I0810 20:39:44.510699 140688498882304 checkpointer.py:236] Initialized all vars.
I0810 20:39:44.513259 140688498882304 executor.py:400] Compiling 1 programs in parallel.
I0810 20:39:44.513607 140688056727296 base_runner.py:120] Init inputs TrainProgram
I0810 20:39:44.513836 140688056727296 base_runner.py:120] Init inputs TrainProgram done.
I0810 20:39:44.513968 140688056727296 base_runner.py:120] Compiling TrainProgram
I0810 20:41:20.058200 140688498882304 executor.py:422] Retrieve params.
I0810 20:41:20.058756 140688498882304 executor.py:424] Retrieve params done.
I0810 20:41:20.058939 140688498882304 checkpointer.py:181] Save checkpoint
2021-08-10 20:51:18.167557: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:159] RPC failed with status = "Unavai
lable: Socket closed" and grpc_error_string = "{"created":"@1628628678.167318634","description":"Error received from peer ipv4:10
.70.9.74:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed
","grpc_status":14}", maybe retrying the RPC
Exception in thread SessionCloseThread:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/usr/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 765, in close
tf_session.TF_CloseSession(self._session)
tensorflow.python.framework.errors_impl.AbortedError: Session 0c4ba843fb27c733 is not found. Possibly, this master has restarted.
I0810 20:51:41.948596 140688498882304 base_runner.py:120] Job : Retrying as expected
executor_tpu exception: Session 0c4ba843fb27c733 is not found.
I0810 20:51:42.952887 140688498882304 retry.py:62] Retry: caught exception: _RunLoop while running tensorflow.python.framework.er
rors_impl.AbortedError: Session 0c4ba843fb27c733 is not found.
. Call failed at (most recent call last):
File "/usr/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/usr/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
Traceback for above exception (most recent call last):
File "/root/.cache/bazel/_bazel_root/17eb95f0bc03547f4f1319e61997e114/execroot/main/bazel-out/k8-opt/bin/lingvo/trainer.run
files/main/lingvo/core/retry.py", line 49, in Wrapper
return func(*args, **kwargs)
File "/root/.cache/bazel/_bazel_root/17eb95f0bc03547f4f1319e61997e114/execroot/main/bazel-out/k8-opt/bin/lingvo/trainer.run
files/main/lingvo/base_runner.py", line 228, in _RunLoop
loop_func(*loop_args)
File "/root/.cache/bazel/_bazel_root/17eb95f0bc03547f4f1319e61997e114/execroot/main/bazel-out/k8-opt/bin/lingvo/trainer.run
files/main/lingvo/executor.py", line 441, in _Loop
RunSave(sess, global_step)
File "/root/.cache/bazel/_bazel_root/17eb95f0bc03547f4f1319e61997e114/execroot/main/bazel-out/k8-opt/bin/lingvo/trainer.run
files/main/lingvo/executor.py", line 430, in RunSave
self.save_only_checkpointer.Save(sess, global_step)
File "/root/.cache/bazel/_bazel_root/17eb95f0bc03547f4f1319e61997e114/execroot/main/bazel-out/k8-opt/bin/lingvo/trainer.run
files/main/lingvo/core/checkpointer.py", line 182, in Save
path = self._saver.save(sess, self._save_path, gsteps)
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/training/saver.py", line 1188, in save
model_checkpoint_path = sess.run(
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
Waiting for 1.53 seconds before retrying.
I0810 20:51:42.953114 140688498882304 base_runner.py:227] executor_tpu started.

Lingua principale
Python
Stelle
2.9k
Fork
452
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tensorflow/lingvo

Tutte le issue di tensorflow/lingvo

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.