Table not initialized when serving model
@varshaan 已经在做这个了。
开始于 2021年4月27日。
评估
这个 Issue 还没有评估数据。
描述
Posting for @awadalaa
We are blocked on experimenting with a new Tensorflow model in production because it fails to inference with this error:
tensorflow.python.framework.errors_impl.FailedPreconditionError: Table not initialized.
We have narrowed down the issue to a bit of our code that applies a bm25 transformation in a Tensorflow-Transform job. As part of applying that transformation, it learns and applies a vocabulary however when we inference the model it fails to initialize the table from that vocabulary file on this line. Here is the BM25 code we are using and the line where it fails:
https://gist.github.com/awadalaa/e9290cf6674884d8e197fe315ed7d832#file-gistfile1-txt-L176-L177
More background:
We run a Tensorflow-Transform Beam/Dataflow job that executes this transformation and saves the transform graph. Later when we train our model, we save it with a signature that applies the TFT layer: transformed_features = model.tft_layer(parsed_features). We noticed that the exported model/assets directory does not include the intermediate vocabulary used by the above BM25 transformation although it does include every other vocabulary file learned in the TFT job. Any ideas why the above transformation would fail to export the vocabulary assets for a saved model?
Stack trace here:
Traceback (most recent call last):
File "/Users/aawad/Desktop/keras_predict.py", line 174, in <module>
print("prediction_output", predict(inference_data))
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1655, in __call__
return self._call_impl(args, kwargs)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1673, in _call_impl
return self._call_with_flat_signature(args, kwargs, cancellation_manager)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1722, in _call_with_flat_signature
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 106, in _call_flat
cancellation_manager)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
ctx=ctx)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Table not initialized.
[[{{node StatefulPartitionedCall/StatefulPartitionedCall/transform_features_layer/StatefulPartitionedCall/transform/apply_haystack_vocabulary_query_ngram_substrings_tags_ngram_substrings/hash_table_Lookup/LookupTableFindV2}}]] [Op:__inference_signature_wrapper_23443]
Function call stack:
signature_wrapper
- 主要语言
- Python
- 星标
- 989
- 派生
- 225
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
tensorflow/transform 的其他 Issue
-
stat:contributions welcome type:bug
难度 2/5 1-3 小时 新手友好度 45/100
tensorflow/transform#347 ·
-
bug stat:awaiting response
难度 3/5 1-2 天 新手友好度 35/100
tensorflow/transform#339 · 2 条评论 ·
-
examples/README.md has dead link to getting started可能重新可做 @pindinagesh 于 1599 天前认领,目前没有进行中的 PR。 未关闭stat:contributions welcome type:support
tensorflow/transform#272 · 4 条评论 · 已指派 1 人 ·
-
scale_to_z_score_per_key should give caller control over OOV behavior可能重新可做 @iindyk 于 1791 天前认领,目前没有进行中的 PR。 未关闭stat:contributions welcome type:feature
tensorflow/transform#252 · 6 条评论 · 已指派 2 人 ·
-
tft.apply_saved_model raises ValueError when running beam pipeline可能重新可做 @varshaan 于 2006 天前认领,目前没有进行中的 PR。 未关闭stat:awaiting tensorflower type:support
tensorflow/transform#231 · 7 条评论 · 已指派 2 人 ·
查看 tensorflow/transform 的全部 Issue
相似的 Issue
-
needs triage
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
json_params_matcher fails on falsy top-level JSON primitives (0, False, "")可能已有人在做 @mayureshsonawane17 今天认领。 未关闭Waiting for: Product Owner
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 5 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 88/100
ayghri/i-have-adhd#249 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
modelscope/FunASR#3762 ·
维护者通常 1 天内回复