Firestore's @transactional is not thread safe
メンテナーはふだん 2 日以内に返信
評価
調査の方向性
The reproduction is in main.py and uses firestore.transactional with two concurrent Transaction instances. Start by tracing the decorator's retry handling and where retry transaction IDs are stored; then check the related Firestore transaction tests. Done when concurrent decorated calls keep their retry IDs isolated and a regression test reproduces the race.
索引モデルが issue の本文から書いたものです。
説明
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
Context
I use @firestore.transactional to run transactions in python
Expected Behavior:
I'd expect the wrapper to be thread safe and allow multiple concurrent calls since it's a method decorator.
Actual Behavior:
On retries, transactions end up with each other's retry ids, which causes failures.
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 276, in __call__
response, ignored_call = self._with_call(
^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 331, in _with_call
return call.result(), call
^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 447, in result
raise self
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 314, in continuation
response, call = self._thunk(new_method).with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 77, in with_call
return self._interceptor.intercept_unary(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 177, in intercept_unary
return self._intercept(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 158, in _intercept
raise exc
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 144, in _intercept
result = invoker(request, metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 67, in invoker
return self._base_callable.with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 1182, in with_call
return _end_unary_response_blocking(state, call, True, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 999, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "The referenced transaction has expired or is no longer valid."
debug_error_string = "INVALID_ARGUMENT:The referenced transaction has expired or is no longer valid."
API client name and version
google-cloud-firestore 2.34.1
Reproduction steps: code
file: main.py
"""Repro: concurrent calls of one @firestore.transactional function share its retry id.
Call A's commit is aborted while call B is in flight. A retries with
BeginTransaction(retry_transaction=<B's id>), which in production invalidates B, whose commit then
fails with "The referenced transaction has expired or is no longer valid".
pip install google-cloud-firestore
python transactional_race_repro.py
"""
import threading
from concurrent.futures import ThreadPoolExecutor
from google.api_core import exceptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import firestore
WAIT_SECONDS = 5
class FakeRpcTransaction(firestore.Transaction):
def __init__(self, name, aborted_commits):
super().__init__(
firestore.Client(project="repro", credentials=AnonymousCredentials())
)
self.name = name
self.aborted_commits = aborted_commits
self.begun_with_retry_ids = []
def _begin(self, retry_id=None):
self.begun_with_retry_ids.append(retry_id)
self._id = f"{self.name}-{len(self.begun_with_retry_ids)}".encode()
def _commit(self):
if self.aborted_commits > 0:
self.aborted_commits -= 1
raise exceptions.Aborted("Contention")
self._clean_up()
return []
def _rollback(self):
self._clean_up()
@firestore.transactional
def update_in_transaction(transaction, began, wait_for):
# Release the GIL, as a network call in a transaction would
began.set()
assert wait_for.wait(WAIT_SECONDS)
def main():
a = FakeRpcTransaction("A", aborted_commits=1)
b = FakeRpcTransaction("B", aborted_commits=0)
a_began, b_began, a_finished = (threading.Event() for _ in range(3))
with ThreadPoolExecutor(max_workers=2) as executor:
# 1. A begins and is in flight.
a_call = executor.submit(update_in_transaction, a, a_began, b_began)
assert a_began.wait(WAIT_SECONDS)
# 2. B begins while A is in flight, resetting the shared retry id and then setting it to B's.
b_call = executor.submit(update_in_transaction, b, b_began, a_finished)
# 3. A's commit aborts, so A retries with the shared retry id, which is now B's transaction id.
a_call.result(timeout=WAIT_SECONDS)
a_finished.set()
b_call.result(timeout=WAIT_SECONDS)
print(f"google-cloud-firestore {firestore.__version__}")
print(f"A began with retry ids {a.begun_with_retry_ids}")
assert a.begun_with_retry_ids == [None, b"A-1"], (
f"A's retry began as a retry of B's transaction {a.begun_with_retry_ids[1]!r}, "
"not of its own b'A-1'"
)
if __name__ == "__main__":
main()
Reproduction steps: supporting files
file: mydata.csv
alpha,1,3
beta,2,5
Reproduction steps: actual results
file: output.txtmydata.csv
Calculated: foo
Reproduction steps: expected results
file: output.txtmydata.csv
Calculated: bar
OS & version + platform
OSX
Python environment
3.12.14
Python dependencies
No response
Additional context
No response
- 主要言語
- Python
- スター
- 5.4k
- フォーク
- 1.8k
- 平均マージ
- 2日 20時間
- マージ済み PR(30日)
- 140
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
googleapis/google-cloud-python のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
googleapis/google-cloud-python#19099 ·
メンテナーはふだん 2 日以内に返信
-
firestore: field path with a quoted name ending in a backslash is misparsed対応中かも @sebxt が 28 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
googleapis/google-cloud-python#18771 ·
メンテナーはふだん 2 日以内に返信
-
api: spanner type: feature request
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
googleapis/google-cloud-python#18584 ·
メンテナーはふだん 2 日以内に返信
-
auth: `impersonated_credentials` loses `subject` on copy, so `with_scopes()` and `with_quota_project()` turn off domain wide delegation対応中かも @Om-singhaI が 57 日前に担当しました。 オープンauth priority: p2 type: bug
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
googleapis/google-cloud-python#18428 ·
メンテナーはふだん 2 日以内に返信
-
A client called `run_in_transaction` be stuck if Spanner is down対応中かも @founderinaction が 10 日前に担当しました。 オープンpriority: p2 type: bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
googleapis/google-cloud-python#18375 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
googleapis/google-cloud-python の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
btclib-org/btclib-wallet#1045 ·
メンテナーはふだん 1 日以内に返信
-
area-templates triage:bot-seen triage:needs-human
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
microsoft/aspire#22356 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
[Train] An elastic ScalingConfig(num_workers=(0, 0)) is accepted and the run waits forever対応中かも @wguo7 が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
ray-project/ray#68217 ·
メンテナーはふだん 2 日以内に返信