Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string

Aperta
  • Qualcosa non va?
#238 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python, tensorflow
Ambito
machine-learning

Direzione di ricerca

Start by reproducing the Librispeech960Wpm run with the reported Lingvo 0.7.2 and TensorFlow GPU 2.2.0 setup, then inspect the decoder TensorArrayV2Write error and the AssertSameDim0 GPU kernel failure in the log. The issue does not identify source files or tests, so trace those operations from the training and evaluation entry points. Done means the run completes evaluation without either reported error.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

I am training a Lingvo ASR model from scratch on the Librispeech 100 dataset, where Lingvo is installed via pip. The training environment uses a single GPU. I am running the following command to start the training job:

python3 -m lingvo.trainer \
--run_locally=gpu \
--mode=sync  \
--logdir=/tmp/logdir \
--job=controller,trainer_client,decoder_test,decoder_dev,evaler_test,evaler_dev \
--model=librispeech.Librispeech960Wpm \
--decoder_jobs=dev,test \
--logtostderr \
--enable_asserts=false

This job will then stop with an error message and exit the training job. A short excerpt from the tail of the log looks like this (a longer excerpt can be found at the end of this comment):

I0916 07:50:28.075007 139688576808704 base_runner.py:111] <b>Retrying as expected</b> decoder_test exception: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]

I would be very happy to learn more about this error message as I couldn't find anything that relates to it.

I am using the following environment:

  • RHEL 7.7
  • Python 3.7.9
  • lingvo==0.7.2 and tensorflow-gpu==2.2.0
  • 1x Tesla V100 GPU
  • CUDA 10.1
I0916 07:50:02.547942 139689130432256 base_runner.py:111] step:     0
2020-09-16 07:50:04.789707: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-09-16 07:50:05.167668: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:05.168990: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:05.169057: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:05.169713: I lingvo/core/ops/record_yielder.cc:376] 0x7f0229275fb0 Record yielder start
2020-09-16 07:50:05.169734: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:05.169755: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:05.169781: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/dev-clean.tfrecords-00000-of-00001
2020-09-16 07:50:05.170354: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:05.171319: I lingvo/core/ops/record_yielder.cc:376] 0x7f01699aefb0 Record yielder start
2020-09-16 07:50:05.171354: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:05.171373: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:05.171385: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/test-clean.tfrecords-00000-of-00001
2020-09-16 07:50:05.425343: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:05.520665: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:05.521776: I lingvo/core/ops/record_yielder.cc:376] 0x7f0babaf52f0 Record yielder start
2020-09-16 07:50:05.521795: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:05.521815: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:05.521840: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/test-clean.tfrecords-00000-of-00001
2020-09-16 07:50:05.566739: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:05.568129: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:05.569379: I lingvo/core/ops/record_yielder.cc:376] 0x7f0ba3cb2f90 Record yielder start
2020-09-16 07:50:05.569401: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:05.569420: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:05.569435: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/dev-clean.tfrecords-00000-of-00001
WARNING:tensorflow:Issue encountered when serializing __batch_norm_update_dict.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'dict' object has no attribute 'name'
W0916 07:50:05.692504 139689138824960 meta_graph.py:437] Issue encountered when serializing __batch_norm_update_dict.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
'dict' object has no attribute 'name'
I0916 07:50:06.058790 139689138824960 checkpointer.py:118] Save checkpoint done: /tmp/logdir/train/ckpt-00000000
2020-09-16 07:50:08.329259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-16 07:50:12.957611: I lingvo/core/ops/record_yielder.cc:532] Epoch 1: total records 477
2020-09-16 07:50:12.957696: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/test-clean.tfrecords-00000-of-00001
2020-09-16 07:50:12.958901: I lingvo/core/ops/record_yielder.cc:407] 0x7f0babaf52f0Basic record yielder exit
2020-09-16 07:50:13.165600: I lingvo/core/ops/record_yielder.cc:532] Epoch 1: total records 522
2020-09-16 07:50:13.165677: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/dev-clean.tfrecords-00000-of-00001
2020-09-16 07:50:13.166971: I lingvo/core/ops/record_yielder.cc:407] 0x7f0ba3cb2f90Basic record yielder exit
I0916 07:50:13.276062 139688576808704 base_runner.py:111] <b>Retrying as expected</b> decoder_test exception: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]

2020-09-16 07:50:13.431772: I lingvo/core/ops/record_yielder.cc:532] Epoch 1: total records 464
2020-09-16 07:50:13.431881: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/dev-clean.tfrecords-00000-of-00001
2020-09-16 07:50:13.433208: I lingvo/core/ops/record_yielder.cc:407] 0x7f0229275fb0Basic record yielder exit
I0916 07:50:13.525660 139688568416000 base_runner.py:111] <b>Retrying as expected</b> decoder_dev exception: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]

2020-09-16 07:50:13.655583: I lingvo/core/ops/record_yielder.cc:532] Epoch 1: total records 549
2020-09-16 07:50:13.655646: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/test-clean.tfrecords-00000-of-00001
2020-09-16 07:50:13.656922: I lingvo/core/ops/record_yielder.cc:407] 0x7f01699aefb0Basic record yielder exit
E0916 07:50:13.759514 139688551630592 base_runner.py:251] evaler_dev done (fatal error): <class 'tensorflow.python.framework.errors_impl.NotFoundError'>
I0916 07:50:13.759727 139688551630592 base_runner.py:111] evaler_dev exception: From /job:localhost/replica:0/task:0:
No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
	.  Registered:  device='CPU'

	 [[AssertSameDim0]]
	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_MgCk22KtlSc]]

E0916 07:50:13.760045 139688551630592 base_runner.py:258] Traceback (most recent call last):
E0916 07:50:13.760113 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
E0916 07:50:13.760184 139688551630592 base_runner.py:258]     return fn(*args)
E0916 07:50:13.760227 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
E0916 07:50:13.760267 139688551630592 base_runner.py:258]     target_list, run_metadata)
E0916 07:50:13.760307 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
E0916 07:50:13.760347 139688551630592 base_runner.py:258]     run_metadata)
E0916 07:50:13.760385 139688551630592 base_runner.py:258] tensorflow.python.framework.errors_impl.NotFoundError: From /job:localhost/replica:0/task:0:
E0916 07:50:13.760424 139688551630592 base_runner.py:258] No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
E0916 07:50:13.760462 139688551630592 base_runner.py:258] 	.  Registered:  device='CPU'
E0916 07:50:13.760500 139688551630592 base_runner.py:258] 
E0916 07:50:13.760538 139688551630592 base_runner.py:258] 	 [[AssertSameDim0]]
E0916 07:50:13.760575 139688551630592 base_runner.py:258] 	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_MgCk22KtlSc]]
E0916 07:50:13.760614 139688551630592 base_runner.py:258] 
E0916 07:50:13.760651 139688551630592 base_runner.py:258] During handling of the above exception, another exception occurred:
E0916 07:50:13.760689 139688551630592 base_runner.py:258] 
E0916 07:50:13.760726 139688551630592 base_runner.py:258] Traceback (most recent call last):
E0916 07:50:13.760764 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/base_runner.py", line 193, in _RunLoop
E0916 07:50:13.760802 139688551630592 base_runner.py:258]     loop_func(*loop_args)
E0916 07:50:13.760840 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1001, in _Loop
E0916 07:50:13.760909 139688551630592 base_runner.py:258]     if not path or self._EvalOnce(path, sess):
E0916 07:50:13.760953 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1070, in _EvalOnce
E0916 07:50:13.760994 139688551630592 base_runner.py:258]     ans = sess.run(self._model_task.eval_metrics)
E0916 07:50:13.761034 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
E0916 07:50:13.761075 139688551630592 base_runner.py:258]     run_metadata_ptr)
E0916 07:50:13.761115 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run
E0916 07:50:13.761156 139688551630592 base_runner.py:258]     feed_dict_tensor, options, run_metadata)
E0916 07:50:13.761229 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
E0916 07:50:13.761267 139688551630592 base_runner.py:258]     run_metadata)
E0916 07:50:13.761305 139688551630592 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
E0916 07:50:13.761343 139688551630592 base_runner.py:258]     raise type(e)(node_def, op, message)
E0916 07:50:13.761386 139688551630592 base_runner.py:258] tensorflow.python.framework.errors_impl.NotFoundError: From /job:localhost/replica:0/task:0:
E0916 07:50:13.761426 139688551630592 base_runner.py:258] No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
E0916 07:50:13.761465 139688551630592 base_runner.py:258] 	.  Registered:  device='CPU'
E0916 07:50:13.761503 139688551630592 base_runner.py:258] 
E0916 07:50:13.761541 139688551630592 base_runner.py:258] 	 [[AssertSameDim0]]
E0916 07:50:13.761579 139688551630592 base_runner.py:258] 	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_MgCk22KtlSc]]
E0916 07:50:13.761616 139688551630592 base_runner.py:258] 
E0916 07:50:13.956376 139688560023296 base_runner.py:251] evaler_test done (fatal error): <class 'tensorflow.python.framework.errors_impl.NotFoundError'>
I0916 07:50:13.956582 139688560023296 base_runner.py:111] evaler_test exception: From /job:localhost/replica:0/task:0:
No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
	.  Registered:  device='CPU'

	 [[AssertSameDim0]]
	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_5uVLZBAakkA]]

E0916 07:50:13.956852 139688560023296 base_runner.py:258] Traceback (most recent call last):
E0916 07:50:13.956946 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
E0916 07:50:13.956998 139688560023296 base_runner.py:258]     return fn(*args)
E0916 07:50:13.957043 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
E0916 07:50:13.957085 139688560023296 base_runner.py:258]     target_list, run_metadata)
E0916 07:50:13.957128 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
E0916 07:50:13.957170 139688560023296 base_runner.py:258]     run_metadata)
E0916 07:50:13.957241 139688560023296 base_runner.py:258] tensorflow.python.framework.errors_impl.NotFoundError: From /job:localhost/replica:0/task:0:
E0916 07:50:13.957281 139688560023296 base_runner.py:258] No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
E0916 07:50:13.957319 139688560023296 base_runner.py:258] 	.  Registered:  device='CPU'
E0916 07:50:13.957358 139688560023296 base_runner.py:258] 
E0916 07:50:13.957397 139688560023296 base_runner.py:258] 	 [[AssertSameDim0]]
E0916 07:50:13.957435 139688560023296 base_runner.py:258] 	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_5uVLZBAakkA]]
E0916 07:50:13.957474 139688560023296 base_runner.py:258] 
E0916 07:50:13.957512 139688560023296 base_runner.py:258] During handling of the above exception, another exception occurred:
E0916 07:50:13.957551 139688560023296 base_runner.py:258] 
E0916 07:50:13.957589 139688560023296 base_runner.py:258] Traceback (most recent call last):
E0916 07:50:13.957627 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/base_runner.py", line 193, in _RunLoop
E0916 07:50:13.957665 139688560023296 base_runner.py:258]     loop_func(*loop_args)
E0916 07:50:13.957703 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1001, in _Loop
E0916 07:50:13.957741 139688560023296 base_runner.py:258]     if not path or self._EvalOnce(path, sess):
E0916 07:50:13.957779 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1070, in _EvalOnce
E0916 07:50:13.957818 139688560023296 base_runner.py:258]     ans = sess.run(self._model_task.eval_metrics)
E0916 07:50:13.957856 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
E0916 07:50:13.957932 139688560023296 base_runner.py:258]     run_metadata_ptr)
E0916 07:50:13.957977 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run
E0916 07:50:13.958019 139688560023296 base_runner.py:258]     feed_dict_tensor, options, run_metadata)
E0916 07:50:13.958060 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
E0916 07:50:13.958101 139688560023296 base_runner.py:258]     run_metadata)
E0916 07:50:13.958142 139688560023296 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
E0916 07:50:13.958213 139688560023296 base_runner.py:258]     raise type(e)(node_def, op, message)
E0916 07:50:13.958252 139688560023296 base_runner.py:258] tensorflow.python.framework.errors_impl.NotFoundError: From /job:localhost/replica:0/task:0:
E0916 07:50:13.958290 139688560023296 base_runner.py:258] No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
E0916 07:50:13.958328 139688560023296 base_runner.py:258] 	.  Registered:  device='CPU'
E0916 07:50:13.958366 139688560023296 base_runner.py:258] 
E0916 07:50:13.958403 139688560023296 base_runner.py:258] 	 [[AssertSameDim0]]
E0916 07:50:13.958441 139688560023296 base_runner.py:258] 	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_5uVLZBAakkA]]
E0916 07:50:13.958480 139688560023296 base_runner.py:258] 
I0916 07:50:14.278506 139688576808704 retry.py:71] Retry: caught exception: _RunLoop while running tensorflow.python.framework.errors_impl.InvalidArgumentError: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]
. Call failed at (most recent call last):
  File "/home/hesselt/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/home/hesselt/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/hesselt/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
Traceback for above exception (most recent call last):
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/core/retry.py", line 53, in Wrapper
    return func(*args, **kwargs)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/base_runner.py", line 193, in _RunLoop
    loop_func(*loop_args)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1205, in _Loop
    if not path or self.DecodeCheckpoint(sess, path):
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1255, in DecodeCheckpoint
    options=run_options)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
    run_metadata_ptr)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
Waiting for 1.54 seconds before retrying.
I0916 07:50:14.278706 139688576808704 base_runner.py:192] decoder_test started.
I0916 07:50:14.527590 139688568416000 retry.py:71] Retry: caught exception: _RunLoop while running tensorflow.python.framework.errors_impl.InvalidArgumentError: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]
. Call failed at (most recent call last):
  File "/home/hesselt/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/home/hesselt/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/hesselt/.pyenv/versions/3.7.9/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
Traceback for above exception (most recent call last):
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/core/retry.py", line 53, in Wrapper
    return func(*args, **kwargs)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/base_runner.py", line 193, in _RunLoop
    loop_func(*loop_args)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1205, in _Loop
    if not path or self.DecodeCheckpoint(sess, path):
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 1255, in DecodeCheckpoint
    options=run_options)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
    run_metadata_ptr)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
Waiting for 1.51 seconds before retrying.
I0916 07:50:14.527779 139688568416000 base_runner.py:192] decoder_dev started.
I0916 07:50:15.116318 139688576808704 checkpointer.py:90] Load from checkpoint /tmp/logdir/train/ckpt-00000000.
INFO:tensorflow:Restoring parameters from /tmp/logdir/train/ckpt-00000000
I0916 07:50:15.116988 139688576808704 saver.py:1293] Restoring parameters from /tmp/logdir/train/ckpt-00000000
I0916 07:50:16.108031 139688568416000 checkpointer.py:90] Load from checkpoint /tmp/logdir/train/ckpt-00000000.
INFO:tensorflow:Restoring parameters from /tmp/logdir/train/ckpt-00000000
I0916 07:50:16.108597 139688568416000 saver.py:1293] Restoring parameters from /tmp/logdir/train/ckpt-00000000
I0916 07:50:16.593458 139688576808704 checkpointer.py:92] Load checkpoint done.
I0916 07:50:17.036803 139688568416000 checkpointer.py:92] Load checkpoint done.
I0916 07:50:17.216221 139688576808704 trainer.py:1247] Fetching dec_output.
I0916 07:50:18.367389 139688568416000 trainer.py:1247] Fetching dec_output.
2020-09-16 07:50:19.508041: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:19.510036: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:19.510489: I lingvo/core/ops/record_yielder.cc:376] 0x7f019657b1f0 Record yielder start
2020-09-16 07:50:19.510517: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:19.510536: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:19.510554: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/train/train.tfrecords-*
2020-09-16 07:50:24.602411: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:24.603843: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:24.604663: I lingvo/core/ops/record_yielder.cc:376] 0x7f03c82afb30 Record yielder start
2020-09-16 07:50:24.604685: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:24.604720: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:24.604782: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/dev-clean.tfrecords-00000-of-00001
2020-09-16 07:50:24.835422: I ./lingvo/core/ops/input_common.h:74] Create RecordProcessor
2020-09-16 07:50:24.921652: I lingvo/core/ops/input_common.cc:34] Input source weights are empty, fall back to legacy behavior.
2020-09-16 07:50:24.922524: I lingvo/core/ops/record_yielder.cc:376] 0x7f00f424ccf0 Record yielder start
2020-09-16 07:50:24.922549: I lingvo/core/ops/record_yielder.cc:378] Randomly seed RecordYielder.
2020-09-16 07:50:24.922578: I ./lingvo/core/ops/input_common.h:80] Create batcher
2020-09-16 07:50:24.922608: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/test-clean.tfrecords-00000-of-00001
2020-09-16 07:50:26.498676: I lingvo/core/ops/record_yielder.cc:407] 0x7f019657b1f0Basic record yielder exit
E0916 07:50:27.017492 139689130432256 base_runner.py:251] trainer done (fatal error): <class 'tensorflow.python.framework.errors_impl.NotFoundError'>
I0916 07:50:27.017792 139689130432256 base_runner.py:111] trainer exception: From /job:localhost/replica:0/task:0:
No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
	.  Registered:  device='CPU'

	 [[AssertSameDim0]]
	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_JbKJd4D0nrc]]

E0916 07:50:27.018238 139689130432256 base_runner.py:258] Traceback (most recent call last):
E0916 07:50:27.018307 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
E0916 07:50:27.018360 139689130432256 base_runner.py:258]     return fn(*args)
E0916 07:50:27.018405 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
E0916 07:50:27.018447 139689130432256 base_runner.py:258]     target_list, run_metadata)
E0916 07:50:27.018487 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
E0916 07:50:27.018527 139689130432256 base_runner.py:258]     run_metadata)
E0916 07:50:27.018566 139689130432256 base_runner.py:258] tensorflow.python.framework.errors_impl.NotFoundError: From /job:localhost/replica:0/task:0:
E0916 07:50:27.018604 139689130432256 base_runner.py:258] No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
E0916 07:50:27.018643 139689130432256 base_runner.py:258] 	.  Registered:  device='CPU'
E0916 07:50:27.018682 139689130432256 base_runner.py:258] 
E0916 07:50:27.018721 139689130432256 base_runner.py:258] 	 [[AssertSameDim0]]
E0916 07:50:27.018759 139689130432256 base_runner.py:258] 	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_JbKJd4D0nrc]]
E0916 07:50:27.018798 139689130432256 base_runner.py:258] 
E0916 07:50:27.018836 139689130432256 base_runner.py:258] During handling of the above exception, another exception occurred:
E0916 07:50:27.018901 139689130432256 base_runner.py:258] 
E0916 07:50:27.018945 139689130432256 base_runner.py:258] Traceback (most recent call last):
E0916 07:50:27.018986 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/base_runner.py", line 193, in _RunLoop
E0916 07:50:27.019028 139689130432256 base_runner.py:258]     loop_func(*loop_args)
E0916 07:50:27.019069 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/lingvo/trainer.py", line 481, in _Loop
E0916 07:50:27.019117 139689130432256 base_runner.py:258]     model_task.per_example_tensors,
E0916 07:50:27.019175 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
E0916 07:50:27.019230 139689130432256 base_runner.py:258]     run_metadata_ptr)
E0916 07:50:27.019268 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run
E0916 07:50:27.019307 139689130432256 base_runner.py:258]     feed_dict_tensor, options, run_metadata)
E0916 07:50:27.019345 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
E0916 07:50:27.019383 139689130432256 base_runner.py:258]     run_metadata)
E0916 07:50:27.019420 139689130432256 base_runner.py:258]   File "/home/hesselt/.pyenv/versions/lingvo_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
E0916 07:50:27.019458 139689130432256 base_runner.py:258]     raise type(e)(node_def, op, message)
E0916 07:50:27.019496 139689130432256 base_runner.py:258] tensorflow.python.framework.errors_impl.NotFoundError: From /job:localhost/replica:0/task:0:
E0916 07:50:27.019534 139689130432256 base_runner.py:258] No registered 'AssertSameDim0' OpKernel for 'GPU' devices compatible with node {{node AssertSameDim0}}
E0916 07:50:27.019572 139689130432256 base_runner.py:258] 	.  Registered:  device='CPU'
E0916 07:50:27.019610 139689130432256 base_runner.py:258] 
E0916 07:50:27.019648 139689130432256 base_runner.py:258] 	 [[AssertSameDim0]]
E0916 07:50:27.019686 139689130432256 base_runner.py:258] 	 [[fprop/librispeech/tower_0_0/enc/brnn_L0/Forward_JbKJd4D0nrc]]
E0916 07:50:27.019724 139689130432256 base_runner.py:258] 
2020-09-16 07:50:27.617585: I lingvo/core/ops/record_yielder.cc:532] Epoch 1: total records 167
2020-09-16 07:50:27.617652: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/dev-clean.tfrecords-00000-of-00001
2020-09-16 07:50:27.618559: I lingvo/core/ops/record_yielder.cc:407] 0x7f03c82afb30Basic record yielder exit
I0916 07:50:27.798940 139688568416000 base_runner.py:111] <b>Retrying as expected</b> decoder_dev exception: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]

2020-09-16 07:50:27.901154: I lingvo/core/ops/record_yielder.cc:532] Epoch 1: total records 170
2020-09-16 07:50:27.901218: I lingvo/core/ops/record_yielder.cc:485] Epoch 1 /tmp/librispeech/devtest/test-clean.tfrecords-00000-of-00001
2020-09-16 07:50:27.902120: I lingvo/core/ops/record_yielder.cc:407] 0x7f00f424ccf0Basic record yielder exit
I0916 07:50:28.075007 139688576808704 base_runner.py:111] <b>Retrying as expected</b> decoder_test exception: From /job:localhost/replica:0/task:0:
2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node TensorArrayV2Write/TensorListSetItem}}]]
Lingua principale
Python
Stelle
2.9k
Fork
452
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tensorflow/lingvo

Tutte le issue di tensorflow/lingvo

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.