[Bug] MetricBuffer overflow causes PanicException ("You must initialize the reference before using it") on a later retrieve_updates()

Ouverte
  • Quelque chose ne va pas ?
#2,092 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
58/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend, observability

Piste de recherche

The panic is reported at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14, while the reproduction uses Python's MetricBuffer and retrieve_updates(). Start by tracing how overflow affects the metrics buffer reference and how retrieve_updates() drains it. Done means the buffer can overflow and later drains return updates without panicking, as described in the reproduction.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

What are you really trying to do?

Export SDK and workflow metrics through temporalio.runtime.MetricBuffer, draining it periodically with retrieve_updates() from a background asyncio task.

Describe the bug

When a MetricBuffer fills up between drains, core logs Core's metrics buffer is full! Dropping call to record metrics for each dropped update. That part is expected.

After an overflow:

  1. The first retrieve_updates() returns only part of the updates. In the reproduction below, a MetricBuffer(2000) filled by 300 workflows each adding to one counter returns 695 updates containing 3 of the 300 counter increments.
  2. A later retrieve_updates() raises pyo3_runtime.PanicException: You must initialize the reference before using it, panicking at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14.

PanicException derives from BaseException, so an except Exception around the drain loop does not catch it. The drain task dies, the buffer is never drained again, and every later metric is dropped for the life of the process, with one ERROR log line per dropped update.

Expected: overflow drops updates (as logged), but later drains keep working and return consistent updates, with no panic.

The MetricBuffer docstring describes overflow as a drop plus a logged error:

It is important that the buffer size is set to a high number and that retrieve_updates is called regularly to drain the buffer. If the buffer is full, metric updates will be dropped and an error will be logged.

Minimal Reproduction

The script below uses a small buffer and doesn't drain while the first 300 workflows run, so that it overflows on purpose. We first hit it with MetricBuffer(50_000) drained every 60 seconds, when a burst of workflow activity filled the buffer between two drains.

Run with uv run --with temporalio==1.35.0 python -I repro.py. It uses WorkflowEnvironment.start_local(), so no separate server is needed.

import asyncio

from temporalio import workflow
from temporalio.runtime import MetricBuffer, Runtime, TelemetryConfig
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import UnsandboxedWorkflowRunner, Worker


@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        workflow.metric_meter().create_counter("my_counter").add(1)


async def main() -> None:
    buffer = MetricBuffer(2000)
    runtime = Runtime(telemetry=TelemetryConfig(metrics=buffer))
    async with await WorkflowEnvironment.start_local(runtime=runtime) as env:
        async with Worker(
            env.client,
            task_queue="my-task-queue",
            workflows=[MyWorkflow],
            workflow_runner=UnsandboxedWorkflowRunner(),
        ):

            async def run_workflows(prefix: str, count: int) -> None:
                await asyncio.gather(
                    *[
                        env.client.execute_workflow(
                            MyWorkflow.run,
                            id=f"{prefix}-{i}",
                            task_queue="my-task-queue",
                        )
                        for i in range(count)
                    ]
                )

            await run_workflows("first", 300)
            for attempt in (1, 2):
                updates = buffer.retrieve_updates()
                count = sum(1 for u in updates if u.metric.name == "my_counter")
                print(f"retrieve_updates #{attempt}: {len(updates)} updates, my_counter={count}")
                await run_workflows(f"after-{attempt}", 20)


asyncio.run(main())

Output (the Core's metrics buffer is full! ERROR lines, 8305 in my run, are omitted):

retrieve_updates #1: 695 updates, my_counter=3
thread '<unnamed>' panicked at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14:
...
pyo3_runtime.PanicException: You must initialize the reference before using it

The panic is raised by the second retrieve_updates(). In this script the first 300 workflows run without any drain, and a further 20 workflows run between the two drains.

Environment/Versions
  • OS and processor: macOS arm64
  • Temporal Version: temporalio 1.35.0 (Python SDK, from PyPI); Temporal CLI 1.9.1 dev server (Server 1.32.0) started by WorkflowEnvironment.start_local()
  • Python: 3.14.2
  • Are you using Docker or Kubernetes or building Temporal from source? No, PyPI wheel.
Additional context

None.

Langage dominant
Python
Étoiles
1.2k
Forks
285
Merge moyen
1 j 21 h
PR mergées (30 j)
80

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de temporalio/sdk-python

Toutes les issues de temporalio/sdk-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.