Firestore's @transactional is not thread safe
維護者通常 2 天內回覆
評估
研究方向
The reproduction is in main.py and uses firestore.transactional with two concurrent Transaction instances. Start by tracing the decorator's retry handling and where retry transaction IDs are stored; then check the related Firestore transaction tests. Done when concurrent decorated calls keep their retry IDs isolated and a regression test reproduces the race.
由索引模型根據 Issue 內容生成。
描述
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
Context
I use @firestore.transactional to run transactions in python
Expected Behavior:
I'd expect the wrapper to be thread safe and allow multiple concurrent calls since it's a method decorator.
Actual Behavior:
On retries, transactions end up with each other's retry ids, which causes failures.
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 276, in __call__
response, ignored_call = self._with_call(
^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 331, in _with_call
return call.result(), call
^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 447, in result
raise self
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 314, in continuation
response, call = self._thunk(new_method).with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 77, in with_call
return self._interceptor.intercept_unary(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 177, in intercept_unary
return self._intercept(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 158, in _intercept
raise exc
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 144, in _intercept
result = invoker(request, metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 67, in invoker
return self._base_callable.with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 1182, in with_call
return _end_unary_response_blocking(state, call, True, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 999, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "The referenced transaction has expired or is no longer valid."
debug_error_string = "INVALID_ARGUMENT:The referenced transaction has expired or is no longer valid."
API client name and version
google-cloud-firestore 2.34.1
Reproduction steps: code
file: main.py
"""Repro: concurrent calls of one @firestore.transactional function share its retry id.
Call A's commit is aborted while call B is in flight. A retries with
BeginTransaction(retry_transaction=<B's id>), which in production invalidates B, whose commit then
fails with "The referenced transaction has expired or is no longer valid".
pip install google-cloud-firestore
python transactional_race_repro.py
"""
import threading
from concurrent.futures import ThreadPoolExecutor
from google.api_core import exceptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import firestore
WAIT_SECONDS = 5
class FakeRpcTransaction(firestore.Transaction):
def __init__(self, name, aborted_commits):
super().__init__(
firestore.Client(project="repro", credentials=AnonymousCredentials())
)
self.name = name
self.aborted_commits = aborted_commits
self.begun_with_retry_ids = []
def _begin(self, retry_id=None):
self.begun_with_retry_ids.append(retry_id)
self._id = f"{self.name}-{len(self.begun_with_retry_ids)}".encode()
def _commit(self):
if self.aborted_commits > 0:
self.aborted_commits -= 1
raise exceptions.Aborted("Contention")
self._clean_up()
return []
def _rollback(self):
self._clean_up()
@firestore.transactional
def update_in_transaction(transaction, began, wait_for):
# Release the GIL, as a network call in a transaction would
began.set()
assert wait_for.wait(WAIT_SECONDS)
def main():
a = FakeRpcTransaction("A", aborted_commits=1)
b = FakeRpcTransaction("B", aborted_commits=0)
a_began, b_began, a_finished = (threading.Event() for _ in range(3))
with ThreadPoolExecutor(max_workers=2) as executor:
# 1. A begins and is in flight.
a_call = executor.submit(update_in_transaction, a, a_began, b_began)
assert a_began.wait(WAIT_SECONDS)
# 2. B begins while A is in flight, resetting the shared retry id and then setting it to B's.
b_call = executor.submit(update_in_transaction, b, b_began, a_finished)
# 3. A's commit aborts, so A retries with the shared retry id, which is now B's transaction id.
a_call.result(timeout=WAIT_SECONDS)
a_finished.set()
b_call.result(timeout=WAIT_SECONDS)
print(f"google-cloud-firestore {firestore.__version__}")
print(f"A began with retry ids {a.begun_with_retry_ids}")
assert a.begun_with_retry_ids == [None, b"A-1"], (
f"A's retry began as a retry of B's transaction {a.begun_with_retry_ids[1]!r}, "
"not of its own b'A-1'"
)
if __name__ == "__main__":
main()
Reproduction steps: supporting files
file: mydata.csv
alpha,1,3
beta,2,5
Reproduction steps: actual results
file: output.txtmydata.csv
Calculated: foo
Reproduction steps: expected results
file: output.txtmydata.csv
Calculated: bar
OS & version + platform
OSX
Python environment
3.12.14
Python dependencies
No response
Additional context
No response
- 主要語言
- Python
- 星號
- 5.4k
- 分支
- 1.8k
- 平均合併
- 2 天 20 小時
- 30 天內合併 PR
- 140
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
googleapis/google-cloud-python 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 68/100
googleapis/google-cloud-python#19099 ·
維護者通常 2 天內回覆
-
firestore: field path with a quoted name ending in a backslash is misparsed可能已有人在做 @sebxt 於 28 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 78/100
googleapis/google-cloud-python#18771 ·
維護者通常 2 天內回覆
-
api: spanner type: feature request
難度 2/5 1-3 小時 新手友好度 68/100
googleapis/google-cloud-python#18584 ·
維護者通常 2 天內回覆
-
auth: `impersonated_credentials` loses `subject` on copy, so `with_scopes()` and `with_quota_project()` turn off domain wide delegation可能已有人在做 @Om-singhaI 於 58 天前認領。 未關閉auth priority: p2 type: bug
難度 2/5 1-3 小時 新手友好度 88/100
googleapis/google-cloud-python#18428 ·
維護者通常 2 天內回覆
-
A client called `run_in_transaction` be stuck if Spanner is down可能已有人在做 @founderinaction 於 10 天前認領。 未關閉priority: p2 type: bug
難度 2/5 1-3 小時 新手友好度 72/100
googleapis/google-cloud-python#18375 · 1 則留言 ·
維護者通常 2 天內回覆
查看 googleapis/google-cloud-python 的全部 Issue
相似的 Issue
-
難度 2/5 1-3 小時 新手友好度 64/100
-
難度 2/5 1-3 小時 新手友好度 76/100
btclib-org/btclib-wallet#1045 ·
維護者通常 1 天內回覆
-
area-templates triage:bot-seen triage:needs-human
難度 2/5 1-3 小時 新手友好度 75/100
microsoft/aspire#22356 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 1 天內回覆
-
[Train] An elastic ScalingConfig(num_workers=(0, 0)) is accepted and the run waits forever可能已有人在做 @wguo7 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 75/100
ray-project/ray#68217 ·
維護者通常 2 天內回覆