Firestore's @transactional is not thread safe
I maintainer di solito rispondono entro 2 giorni
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 65/100
Direzione di ricerca
The reproduction is in main.py and uses firestore.transactional with two concurrent Transaction instances. Start by tracing the decorator's retry handling and where retry transaction IDs are stored; then check the related Firestore transaction tests. Done when concurrent decorated calls keep their retry IDs isolated and a regression test reproduces the race.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
Context
I use @firestore.transactional to run transactions in python
Expected Behavior:
I'd expect the wrapper to be thread safe and allow multiple concurrent calls since it's a method decorator.
Actual Behavior:
On retries, transactions end up with each other's retry ids, which causes failures.
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 276, in __call__
response, ignored_call = self._with_call(
^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 331, in _with_call
return call.result(), call
^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 447, in result
raise self
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 314, in continuation
response, call = self._thunk(new_method).with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 77, in with_call
return self._interceptor.intercept_unary(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 177, in intercept_unary
return self._intercept(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 158, in _intercept
raise exc
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 144, in _intercept
result = invoker(request, metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 67, in invoker
return self._base_callable.with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 1182, in with_call
return _end_unary_response_blocking(state, call, True, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 999, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "The referenced transaction has expired or is no longer valid."
debug_error_string = "INVALID_ARGUMENT:The referenced transaction has expired or is no longer valid."
API client name and version
google-cloud-firestore 2.34.1
Reproduction steps: code
file: main.py
"""Repro: concurrent calls of one @firestore.transactional function share its retry id.
Call A's commit is aborted while call B is in flight. A retries with
BeginTransaction(retry_transaction=<B's id>), which in production invalidates B, whose commit then
fails with "The referenced transaction has expired or is no longer valid".
pip install google-cloud-firestore
python transactional_race_repro.py
"""
import threading
from concurrent.futures import ThreadPoolExecutor
from google.api_core import exceptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import firestore
WAIT_SECONDS = 5
class FakeRpcTransaction(firestore.Transaction):
def __init__(self, name, aborted_commits):
super().__init__(
firestore.Client(project="repro", credentials=AnonymousCredentials())
)
self.name = name
self.aborted_commits = aborted_commits
self.begun_with_retry_ids = []
def _begin(self, retry_id=None):
self.begun_with_retry_ids.append(retry_id)
self._id = f"{self.name}-{len(self.begun_with_retry_ids)}".encode()
def _commit(self):
if self.aborted_commits > 0:
self.aborted_commits -= 1
raise exceptions.Aborted("Contention")
self._clean_up()
return []
def _rollback(self):
self._clean_up()
@firestore.transactional
def update_in_transaction(transaction, began, wait_for):
# Release the GIL, as a network call in a transaction would
began.set()
assert wait_for.wait(WAIT_SECONDS)
def main():
a = FakeRpcTransaction("A", aborted_commits=1)
b = FakeRpcTransaction("B", aborted_commits=0)
a_began, b_began, a_finished = (threading.Event() for _ in range(3))
with ThreadPoolExecutor(max_workers=2) as executor:
# 1. A begins and is in flight.
a_call = executor.submit(update_in_transaction, a, a_began, b_began)
assert a_began.wait(WAIT_SECONDS)
# 2. B begins while A is in flight, resetting the shared retry id and then setting it to B's.
b_call = executor.submit(update_in_transaction, b, b_began, a_finished)
# 3. A's commit aborts, so A retries with the shared retry id, which is now B's transaction id.
a_call.result(timeout=WAIT_SECONDS)
a_finished.set()
b_call.result(timeout=WAIT_SECONDS)
print(f"google-cloud-firestore {firestore.__version__}")
print(f"A began with retry ids {a.begun_with_retry_ids}")
assert a.begun_with_retry_ids == [None, b"A-1"], (
f"A's retry began as a retry of B's transaction {a.begun_with_retry_ids[1]!r}, "
"not of its own b'A-1'"
)
if __name__ == "__main__":
main()
Reproduction steps: supporting files
file: mydata.csv
alpha,1,3
beta,2,5
Reproduction steps: actual results
file: output.txtmydata.csv
Calculated: foo
Reproduction steps: expected results
file: output.txtmydata.csv
Calculated: bar
OS & version + platform
OSX
Python environment
3.12.14
Python dependencies
No response
Additional context
No response
- Lingua principale
- Python
- Stelle
- 5.4k
- Fork
- 1.8k
- Merge medio
- 2g 20h
- PR unite (30g)
- 140
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di googleapis/google-cloud-python
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
googleapis/google-cloud-python#19099 ·
I maintainer di solito rispondono entro 2 giorni
-
firestore: field path with a quoted name ending in a backslash is misparsedForse già presa @sebxt l’ha presa 28 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
googleapis/google-cloud-python#18771 ·
I maintainer di solito rispondono entro 2 giorni
-
api: spanner type: feature request
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
googleapis/google-cloud-python#18584 ·
I maintainer di solito rispondono entro 2 giorni
-
auth: `impersonated_credentials` loses `subject` on copy, so `with_scopes()` and `with_quota_project()` turn off domain wide delegationForse già presa @Om-singhaI l’ha presa 57 giorni fa. Apertaauth priority: p2 type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
googleapis/google-cloud-python#18428 ·
I maintainer di solito rispondono entro 2 giorni
-
A client called `run_in_transaction` be stuck if Spanner is downForse già presa @founderinaction l’ha presa 10 giorni fa. Apertapriority: p2 type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
googleapis/google-cloud-python#18375 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
Tutte le issue di googleapis/google-cloud-python
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
btclib-org/btclib-wallet#1045 ·
I maintainer di solito rispondono entro 1 giorno
-
area-templates triage:bot-seen triage:needs-human
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
microsoft/aspire#22356 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno
-
[Train] An elastic ScalingConfig(num_workers=(0, 0)) is accepted and the run waits foreverForse già presa @wguo7 l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
ray-project/ray#68217 ·
I maintainer di solito rispondono entro 2 giorni