[Bug] Dropping any client stops log forwarding for every other client of the same Runtime
Les mainteneurs répondent en général sous 1 jour
Évaluation
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Accessibilité débutants
- 57/100
- Type d'issue
- Bug
- Clarté
- Clairement spécifiée
- Activité
- Active
- Domaine
- backend, observability
Piste de recherche
Start in temporalio/bridge/src/runtime.rs and trace how cloned Runtime values manage log_forwarder_handle during Drop; compare that ownership with the clone created in client.rs. Check the runtime and log forwarding tests for existing lifecycle coverage. Done means dropping one client no longer stops forwarded logs for clients still using the shared runtime, while forwarding stops when the runtime is no longer owned.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
What are you really trying to do?
Forward core logs to a Python logger with LogForwardingConfig, on a Runtime that several clients share. Some clients are short-lived (made for one call and dropped), and one is kept for a worker.
Describe the bug
When any client of a Runtime is dropped, log forwarding stops for every other client of that runtime, for the rest of the process. No error or warning is shown: the forwarded logger just receives nothing more.
The cause appears to be in temporalio/bridge/src/runtime.rs (same on main and in 1.34.0):
Runtimeis#[derive(Clone)], and each client keeps a clone of it (client.rs:let runtime = runtime_ref.runtime.clone();).log_forwarder_handle: Option<Arc<JoinHandle<()>>>is shared by every clone.impl Drop for Runtimecallshandle.abort()on that shared handle.
So the first clone to drop, which is the first client garbage-collected, aborts the forwarder that the RuntimeRef and every remaining client still rely on.
A possible fix: abort the forwarder only when the last owner goes away. For example, wrap the JoinHandle in a small type that aborts in its own Drop, and keep that type behind the Arc. Or implement Drop on RuntimeRef only.
Minimal Reproduction
Against a local dev server (temporal server start-dev):
"""Dropping one client of a Runtime stops log forwarding for every other client of that Runtime."""
import asyncio, gc, logging, sys, uuid
from datetime import timedelta
from temporalio import workflow
from temporalio.client import Client
from temporalio.runtime import LogForwardingConfig, LoggingConfig, Runtime, TelemetryConfig, TelemetryFilter
from temporalio.worker import Worker
@workflow.defn
class Noop:
@workflow.run
async def run(self) -> None:
return None
class Count(logging.Handler):
def __init__(self): super().__init__(); self.n = 0
def emit(self, record): self.n += 1
async def main(drop_one: bool) -> int:
handler = Count()
logger = logging.getLogger(f"forwarded-{uuid.uuid4()}")
logger.addHandler(handler); logger.setLevel(logging.DEBUG); logger.propagate = False
runtime = Runtime(telemetry=TelemetryConfig(logging=LoggingConfig(
filter=TelemetryFilter(core_level="INFO", other_level="WARN"),
forwarding=LogForwardingConfig(logger=logger))))
kept = await Client.connect("localhost:7233", runtime=runtime)
if drop_one:
other = await Client.connect("localhost:7233", runtime=runtime)
del other
gc.collect()
async with Worker(kept, task_queue=f"q-{uuid.uuid4()}", workflows=[Noop]):
await asyncio.sleep(3)
await asyncio.sleep(1)
return handler.n
if __name__ == "__main__":
drop = sys.argv[1] == "drop"
print(("drop one client" if drop else "keep every client"), "->", asyncio.run(main(drop)), "core records forwarded")
Output, run twice each (temporalio 1.32.0, CPython 3.13, macOS arm64):
keep every client -> 2 core records forwarded
drop one client -> 0 core records forwarded
keep every client -> 4 core records forwarded
drop one client -> 0 core records forwarded
The only difference between the two runs is a second client made on the same runtime and dropped before the worker starts. In a longer test, a worker kept alive for 25 s forwarded 18 core records. The same worker after one short-lived client on its runtime was dropped forwarded none. Eight clients kept alive forwarded 15; the same eight dropped forwarded 0.
The workaround is one Runtime per client. It costs a thread per CPU for each runtime.
Environment/Versions
- OS and processor: macOS, Apple Silicon
- Temporal version: Python SDK 1.32.0 reproduced; the same code is in
runtime.rsin 1.34.0 and onmain - Server:
temporal server start-dev(Temporal CLI)
- Langage dominant
- Python
- Étoiles
- 1.2k
- Forks
- 269
- Merge moyen
- 1 j 20 h
- PR mergées (30 j)
- 60
Préparer son environnement
- Aucun Dockerfile ni fichier Docker Compose
- Aucun modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de temporalio/sdk-python
-
bug
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
temporalio/sdk-python#496 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 5/5 Plus d'une semaine Accessibilité débutants 8/100
temporalio/sdk-python#1976 · 1 commentaire ·
Les mainteneurs répondent en général sous 1 jour
-
Proposal: handle SIGTERM by default to initiate graceful worker shutdownPeut-être pris @eamsden l’a pris il y a 12 jours. Ouverte
temporalio/sdk-python#1948 · 1 personne assignée ·
Les mainteneurs répondent en général sous 1 jour
-
Cloud CI Skips Nexus TestsPeut-être à nouveau libre @tconley1428 l’a pris il y a 70 jours, et aucune pull request n’est ouverte. Ouverte
temporalio/sdk-python#1704 · 1 personne assignée ·
Les mainteneurs répondent en général sous 1 jour
-
Windows ARM64 wheel supportOuverte
Difficulté 4/5 3-5 jours Accessibilité débutants 45/100
temporalio/sdk-python#1592 · 8 commentaires ·
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de temporalio/sdk-python
Issues similaires
-
Difficulté 1/5 Moins d'une heure Accessibilité débutants 80/100
521xueweihan/HelloGitHub#4063 ·
-
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
-
Difficulté 2/5 1-3 heures Accessibilité débutants 62/100
Vanilla-OS/apx-gui#112 ·
-
Devstral and Voxtral: section 5 fails in us-east-1 with 400 "Model does not support audio modality"Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
aws-samples/sample-per-model-bedrock#409 ·
Les mainteneurs répondent en général sous 1 jour
-
DuckDB 2.0 breaks to_pyarrow_batchesPeut-être pris @DG47 l’a pris aujourd’hui. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
ibis-project/ibis#12196 ·
Les mainteneurs répondent en général sous 1 jour