[Bug] MetricBuffer overflow causes PanicException ("You must initialize the reference before using it") on a later retrieve_updates()

Abierto
  • ¿Algo no está bien?
#2,092 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
58/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, observability

Línea de trabajo

The panic is reported at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14, while the reproduction uses Python's MetricBuffer and retrieve_updates(). Start by tracing how overflow affects the metrics buffer reference and how retrieve_updates() drains it. Done means the buffer can overflow and later drains return updates without panicking, as described in the reproduction.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What are you really trying to do?

Export SDK and workflow metrics through temporalio.runtime.MetricBuffer, draining it periodically with retrieve_updates() from a background asyncio task.

Describe the bug

When a MetricBuffer fills up between drains, core logs Core's metrics buffer is full! Dropping call to record metrics for each dropped update. That part is expected.

After an overflow:

  1. The first retrieve_updates() returns only part of the updates. In the reproduction below, a MetricBuffer(2000) filled by 300 workflows each adding to one counter returns 695 updates containing 3 of the 300 counter increments.
  2. A later retrieve_updates() raises pyo3_runtime.PanicException: You must initialize the reference before using it, panicking at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14.

PanicException derives from BaseException, so an except Exception around the drain loop does not catch it. The drain task dies, the buffer is never drained again, and every later metric is dropped for the life of the process, with one ERROR log line per dropped update.

Expected: overflow drops updates (as logged), but later drains keep working and return consistent updates, with no panic.

The MetricBuffer docstring describes overflow as a drop plus a logged error:

It is important that the buffer size is set to a high number and that retrieve_updates is called regularly to drain the buffer. If the buffer is full, metric updates will be dropped and an error will be logged.

Minimal Reproduction

The script below uses a small buffer and doesn't drain while the first 300 workflows run, so that it overflows on purpose. We first hit it with MetricBuffer(50_000) drained every 60 seconds, when a burst of workflow activity filled the buffer between two drains.

Run with uv run --with temporalio==1.35.0 python -I repro.py. It uses WorkflowEnvironment.start_local(), so no separate server is needed.

import asyncio

from temporalio import workflow
from temporalio.runtime import MetricBuffer, Runtime, TelemetryConfig
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import UnsandboxedWorkflowRunner, Worker


@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self) -> None:
        workflow.metric_meter().create_counter("my_counter").add(1)


async def main() -> None:
    buffer = MetricBuffer(2000)
    runtime = Runtime(telemetry=TelemetryConfig(metrics=buffer))
    async with await WorkflowEnvironment.start_local(runtime=runtime) as env:
        async with Worker(
            env.client,
            task_queue="my-task-queue",
            workflows=[MyWorkflow],
            workflow_runner=UnsandboxedWorkflowRunner(),
        ):

            async def run_workflows(prefix: str, count: int) -> None:
                await asyncio.gather(
                    *[
                        env.client.execute_workflow(
                            MyWorkflow.run,
                            id=f"{prefix}-{i}",
                            task_queue="my-task-queue",
                        )
                        for i in range(count)
                    ]
                )

            await run_workflows("first", 300)
            for attempt in (1, 2):
                updates = buffer.retrieve_updates()
                count = sum(1 for u in updates if u.metric.name == "my_counter")
                print(f"retrieve_updates #{attempt}: {len(updates)} updates, my_counter={count}")
                await run_workflows(f"after-{attempt}", 20)


asyncio.run(main())

Output (the Core's metrics buffer is full! ERROR lines, 8305 in my run, are omitted):

retrieve_updates #1: 695 updates, my_counter=3
thread '<unnamed>' panicked at sdk-core/crates/common/src/telemetry/metrics/core.rs:121:14:
...
pyo3_runtime.PanicException: You must initialize the reference before using it

The panic is raised by the second retrieve_updates(). In this script the first 300 workflows run without any drain, and a further 20 workflows run between the two drains.

Environment/Versions
  • OS and processor: macOS arm64
  • Temporal Version: temporalio 1.35.0 (Python SDK, from PyPI); Temporal CLI 1.9.1 dev server (Server 1.32.0) started by WorkflowEnvironment.start_local()
  • Python: 3.14.2
  • Are you using Docker or Kubernetes or building Temporal from source? No, PyPI wheel.
Additional context

None.

Lenguaje dominante
Python
Estrellas
1.2k
Forks
285
Merge medio
1 d 21 h
PR fusionados (30 d)
80

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de temporalio/sdk-python

Todos los issues de temporalio/sdk-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.