Table not initialized when serving model
@varshaan ci sta già lavorando.
Dal 27/4/2021.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Posting for @awadalaa
We are blocked on experimenting with a new Tensorflow model in production because it fails to inference with this error:
tensorflow.python.framework.errors_impl.FailedPreconditionError: Table not initialized.
We have narrowed down the issue to a bit of our code that applies a bm25 transformation in a Tensorflow-Transform job. As part of applying that transformation, it learns and applies a vocabulary however when we inference the model it fails to initialize the table from that vocabulary file on this line. Here is the BM25 code we are using and the line where it fails:
https://gist.github.com/awadalaa/e9290cf6674884d8e197fe315ed7d832#file-gistfile1-txt-L176-L177
More background:
We run a Tensorflow-Transform Beam/Dataflow job that executes this transformation and saves the transform graph. Later when we train our model, we save it with a signature that applies the TFT layer: transformed_features = model.tft_layer(parsed_features). We noticed that the exported model/assets directory does not include the intermediate vocabulary used by the above BM25 transformation although it does include every other vocabulary file learned in the TFT job. Any ideas why the above transformation would fail to export the vocabulary assets for a saved model?
Stack trace here:
Traceback (most recent call last):
File "/Users/aawad/Desktop/keras_predict.py", line 174, in <module>
print("prediction_output", predict(inference_data))
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1655, in __call__
return self._call_impl(args, kwargs)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1673, in _call_impl
return self._call_with_flat_signature(args, kwargs, cancellation_manager)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1722, in _call_with_flat_signature
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 106, in _call_flat
cancellation_manager)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
ctx=ctx)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Table not initialized.
[[{{node StatefulPartitionedCall/StatefulPartitionedCall/transform_features_layer/StatefulPartitionedCall/transform/apply_haystack_vocabulary_query_ngram_substrings_tags_ngram_substrings/hash_table_Lookup/LookupTableFindV2}}]] [Op:__inference_signature_wrapper_23443]
Function call stack:
signature_wrapper
- Lingua principale
- Python
- Stelle
- 992
- Fork
- 226
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di tensorflow/transform
-
bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
tensorflow/transform#354 ·
-
stat:contributions welcome type:bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 45/100
tensorflow/transform#347 ·
-
scale_to_z_score_per_key should give caller control over OOV behaviorForse di nuovo libera @iindyk l’ha presa 1838 giorni fa e non c’è nessuna pull request aperta. Apertastat:contributions welcome type:feature
tensorflow/transform#252 · 6 commenti · 2 assegnatari ·
-
tft.apply_saved_model raises ValueError when running beam pipelineForse di nuovo libera @varshaan l’ha presa 2053 giorni fa e non c’è nessuna pull request aperta. Apertastat:awaiting tensorflower type:support
tensorflow/transform#231 · 7 commenti · 2 assegnatari ·
Tutte le issue di tensorflow/transform
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 73/100
I maintainer di solito rispondono entro 1 giorno
-
Unused loot bag group names left in dreamcraft en_US.langForse già presa Una pull request collegata a questa issue è aperta o già unita. ApertaStatus: Triage Suggestion
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 90/100
GTNewHorizons/GT-New-Horizons-Modpack#27789 ·
I maintainer di solito rispondono entro 1 giorno
-
bug frontend sev-5
Difficoltà 2/5 1-3 ore Idoneità per principianti 83/100
-
core good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
-
b4b code health next
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
ESCOMP/CTSM#4330 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno