NotFoundError: Graph execution error: TPU
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 28/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python, tensorflow
- Domain
- machine-learning
Research direction
Start by reproducing the reported model_.fit failure with Python 3.10, TensorFlow 2.15, Keras 3.0.5, and a local TPU VM using the supplied MNIST example. Inspect the complete TPU graph-execution error and determine whether the failure is in the TensorFlow/Keras/TPU interaction; done means identifying a reproducible cause and a verified resolution.
Written by the indexing model from the issue text.
Description
While trying to run the following code on tpu-vm, it didn't work.
tf: 2.15
keras: 3.0.5
tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu="local")
strategy = tf.distribute.TPUStrategy(tpu)
def get_compiled_model():
# Make a simple 2-layer densely-connected neural network.
inputs = keras.Input(shape=(784,))
x = keras.layers.Dense(256, activation="relu")(inputs)
x = keras.layers.Dense(256, activation="relu")(x)
outputs = keras.layers.Dense(10)(x)
model = keras.Model(inputs, outputs)
model.compile(
optimizer=keras.optimizers.Adam(),
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[keras.metrics.SparseCategoricalAccuracy()],
)
return model
def get_dataset():
batch_size = 32
num_val_samples = 10000
# Return the MNIST dataset in the form of a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset).
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Preprocess the data (these are Numpy arrays)
x_train = x_train.reshape(-1, 784).astype("float32") / 255
x_test = x_test.reshape(-1, 784).astype("float32") / 255
y_train = y_train.astype("float32")
y_test = y_test.astype("float32")
# Reserve num_val_samples samples for validation
x_val = x_train[-num_val_samples:]
y_val = y_train[-num_val_samples:]
x_train = x_train[:-num_val_samples]
y_train = y_train[:-num_val_samples]
return (
tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(batch_size),
tf.data.Dataset.from_tensor_slices((x_val, y_val)).batch(batch_size),
tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(batch_size),
)
with strategy.scope():
model_ = get_compiled_model()
train_dataset, val_dataset, test_dataset = get_dataset()
model_.fit(train_dataset, epochs=2, validation_data=val_dataset)
---------------------------------------------------------------------------
NotFoundError Traceback (most recent call last)
Cell In[5], line 1
----> 1 model_.fit(train_dataset, epochs=2, validation_data=val_dataset)
File /usr/local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:123, in filter_traceback.<locals>.error_handler(*args, **kwargs)
120 filtered_tb = _process_traceback_frames(e.__traceback__)
121 # To get the full stack trace, call:
122 # `keras.config.disable_traceback_filtering()`
--> 123 raise e.with_traceback(filtered_tb) from None
124 finally:
125 del filtered_tb
File /usr/local/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
51 try:
52 ctx.ensure_initialized()
---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
54 inputs, attrs, num_outputs)
55 except core._NotOkStatusException as e:
56 if name is not None:
NotFoundError: Graph execution error:
Detected at node TPUReplicate/_compile/_9074053372847989778/_4 defined at (most recent call last):
<stack traces unavailable>
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 1k
- Avg merge
- 7h 38m
- Merged PRs (30d)
- 9
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Kaggle/docker-python
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Kaggle/docker-python#1482 · 2 reactions ·
Maintainers usually reply within 1 day
-
loraOpen
Difficulty 5/5 Over a week Newbie friendliness 20/100
Kaggle/docker-python#1564 ·
Maintainers usually reply within 1 day
-
bug help wanted
Difficulty 3/5 1-2 days Newbie friendliness 25/100
Kaggle/docker-python#1563 ·
Maintainers usually reply within 1 day
-
ذOpen
Difficulty 5/5 Over a week Newbie friendliness 1/100
Kaggle/docker-python#1555 ·
Maintainers usually reply within 1 day
-
Pytorch CUDA P100 GPU IncompatibilityPossibly taken @shivansh193 claimed this 76 days ago. Openbug help wanted
Difficulty 4/5 3-5 days Newbie friendliness 42/100
Kaggle/docker-python#1546 · 4 comments · 2 reactions ·
Maintainers usually reply within 1 day
All issues in Kaggle/docker-python
Similar issues
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 74/100
BrainLesion/BraTS#200 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OpenHands/extensions#970 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
rl-institut/enetra#91 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/datasets#9100 ·
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
NVIDIA/garak#2486 · 1 comment ·
Maintainers usually reply within 2 days