[Bug] MetricBuffer overflow causes PanicException ("You must initialize the reference before using it") on a later retrieve_updates()

Offen
  • Stimmt etwas nicht?
#2,092 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
58/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
backend, observability

Rechercherichtung

The panic is reported at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14, while the reproduction uses Python's MetricBuffer and retrieve_updates(). Start by tracing how overflow affects the metrics buffer reference and how retrieve_updates() drains it. Done means the buffer can overflow and later drains return updates without panicking, as described in the reproduction.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

What are you really trying to do?

Export SDK and workflow metrics through temporalio.runtime.MetricBuffer, draining it periodically with retrieve_updates() from a background asyncio task.

Describe the bug

When a MetricBuffer fills up between drains, core logs Core's metrics buffer is full! Dropping call to record metrics for each dropped update. That part is expected.

After an overflow:

  1. The first retrieve_updates() returns only part of the updates. In the reproduction below, a MetricBuffer(2000) filled by 300 workflows each adding to one counter returns 695 updates containing 3 of the 300 counter increments.
  2. A later retrieve_updates() raises pyo3_runtime.PanicException: You must initialize the reference before using it, panicking at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14.

PanicException derives from BaseException, so an except Exception around the drain loop does not catch it. The drain task dies, the buffer is never drained again, and every later metric is dropped for the life of the process, with one ERROR log line per dropped update.

Expected: overflow drops updates (as logged), but later drains keep working and return consistent updates, with no panic.

The MetricBuffer docstring describes overflow as a drop plus a logged error:

It is important that the buffer size is set to a high number and that retrieve_updates is called regularly to drain the buffer. If the buffer is full, metric updates will be dropped and an error will be logged.

Minimal Reproduction

The script below uses a small buffer and doesn't drain while the first 300 workflows run, so that it overflows on purpose. We first hit it with MetricBuffer(50_000) drained every 60 seconds, when a burst of workflow activity filled the buffer between two drains.

Run with uv run --with temporalio==1.35.0 python -I repro.py. It uses WorkflowEnvironment.start_local(), so no separate server is needed.

import asyncio

from temporalio import workflow
from temporalio.runtime import MetricBuffer, Runtime, TelemetryConfig
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import UnsandboxedWorkflowRunner, Worker


@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        workflow.metric_meter().create_counter("my_counter").add(1)


async def main() -> None:
    buffer = MetricBuffer(2000)
    runtime = Runtime(telemetry=TelemetryConfig(metrics=buffer))
    async with await WorkflowEnvironment.start_local(runtime=runtime) as env:
        async with Worker(
            env.client,
            task_queue="my-task-queue",
            workflows=[MyWorkflow],
            workflow_runner=UnsandboxedWorkflowRunner(),
        ):

            async def run_workflows(prefix: str, count: int) -> None:
                await asyncio.gather(
                    *[
                        env.client.execute_workflow(
                            MyWorkflow.run,
                            id=f"{prefix}-{i}",
                            task_queue="my-task-queue",
                        )
                        for i in range(count)
                    ]
                )

            await run_workflows("first", 300)
            for attempt in (1, 2):
                updates = buffer.retrieve_updates()
                count = sum(1 for u in updates if u.metric.name == "my_counter")
                print(f"retrieve_updates #{attempt}: {len(updates)} updates, my_counter={count}")
                await run_workflows(f"after-{attempt}", 20)


asyncio.run(main())

Output (the Core's metrics buffer is full! ERROR lines, 8305 in my run, are omitted):

retrieve_updates #1: 695 updates, my_counter=3
thread '<unnamed>' panicked at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14:
...
pyo3_runtime.PanicException: You must initialize the reference before using it

The panic is raised by the second retrieve_updates(). In this script the first 300 workflows run without any drain, and a further 20 workflows run between the two drains.

Environment/Versions
  • OS and processor: macOS arm64
  • Temporal Version: temporalio 1.35.0 (Python SDK, from PyPI); Temporal CLI 1.9.1 dev server (Server 1.32.0) started by WorkflowEnvironment.start_local()
  • Python: 3.14.2
  • Are you using Docker or Kubernetes or building Temporal from source? No, PyPI wheel.
Additional context

None.

Vorherrschende Sprache
Python
Sterne
1.2k
Forks
285
Ø Merge
1 T. 21 Std.
Gemergte PRs (30 T.)
80

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus temporalio/sdk-python

Alle Issues in temporalio/sdk-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.