[Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation

Aperta
  • Qualcosa non va?
#2,040 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@avanish-garg ci sta già lavorando.

Dal 29/10/2026.

  • #2041 di @avanish-garg — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
aws, grpc, python
Ambito
backend, observability

Direzione di ricerca

Start in temporalio/contrib/aws/lambda_worker/otel.py at apply_defaults, then read the run_worker configuration path and its existing tests. The issue provides a minimal reproduction using _run_worker_internal; run it across repeated invocations and check whether exporter threads and gRPC channels keep accumulating. Done means repeated calls do not grow those resources, including when another provider already owns the global slot.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sdk-python
What are you really trying to do?

Run a Temporal worker on AWS Lambda with tracing, following the README "Observability" example: call otel.apply_defaults(config, OtelOptions()) inside the configure callback passed to run_worker.

Describe the bug

run_worker calls the user's configure callback on every Lambda invocation (its docstring says so), and a warm container keeps the same process alive. apply_defaults builds a new replay-safe TracerProvider with a new BatchSpanProcessor(OTLPSpanExporter) each time (one daemon export thread and one gRPC channel) and then calls set_tracer_provider(). OpenTelemetry only lets the global provider be set once, so from the second invocation on:

  • OpenTelemetry logs Overriding of current TracerProvider is not allowed,
  • the new provider is never installed and never used (OpenTelemetryPlugin takes the global one), and
  • it is never shut down, so its export thread and channel stay alive for the life of the container.

Result: one extra OtelBatchSpanRecordProcessor thread and gRPC channel per warm invocation, forever.

Expected: the documented per-invocation configure + apply_defaults pattern should be safe to run repeatedly, i.e. no growth in threads or channels across warm invocations.

Minimal Reproduction

Drives the real _run_worker_internal handler with a fake connect and a fake worker, so it needs no Lambda and no collector:

import gc, logging, threading

from opentelemetry import trace

from temporalio.common import WorkerDeploymentVersion
from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, otel
from temporalio.contrib.aws.lambda_worker._run_worker import _run_worker_internal, _WorkerDeps
from temporalio.envconfig import ClientConfigProfile

logging.basicConfig(level=logging.WARNING, format="LOG %(levelname)s %(name)s: %(message)s")


class FakeWorker:
    async def run(self):
        return


async def fake_connect(**kwargs):
    return object()


def configure(config: LambdaWorkerConfig) -> None:  # the README example
    config.worker_config["task_queue"] = "tq"
    otel.apply_defaults(config, otel.OtelOptions())


deps = _WorkerDeps(
    connect=fake_connect,
    create_worker=lambda client, **kw: FakeWorker(),
    load_config=lambda: ClientConfigProfile(address="localhost:7233"),
    getenv=lambda k: None,
)
handler = _run_worker_internal(WorkerDeploymentVersion("svc", "v1"), configure, deps)

providers = []
for invocation in range(1, 6):
    handler({}, None)  # one Lambda invocation in a warm container
    gc.collect()
    providers.append(trace.get_tracer_provider())
    n = sum(1 for t in threading.enumerate() if t.name == "OtelBatchSpanRecordProcessor")
    print(f"after invocation {invocation}: OTLP span-export threads alive={n}, total threads={threading.active_count()}")

print("global tracer provider is the same object every time:", len({id(p) for p in providers}) == 1)

Output on main:

LOG WARNING opentelemetry.trace: Overriding of current TracerProvider is not allowed   (x4)
after invocation 1: OTLP span-export threads alive=1, total threads=2
after invocation 2: OTLP span-export threads alive=2, total threads=3
after invocation 3: OTLP span-export threads alive=3, total threads=4
after invocation 4: OTLP span-export threads alive=4, total threads=5
after invocation 5: OTLP span-export threads alive=5, total threads=6
global tracer provider is the same object every time: True
Environment/Versions
  • OS and processor: macOS (Apple Silicon), Python 3.12.14
  • Temporal Version: temporalio main at 2a7d321c (version 1.35.0), with the native bridge from the 1.34.0 wheel; only pure-Python code from main differs. Extras opentelemetry-exporter-otlp-proto-grpc and opentelemetry-sdk-extension-aws installed.
  • Docker or Kubernetes or building Temporal from source? No. This is a simulated invocation loop; I have not run it on a real Lambda or against a real collector.
Additional context

The relevant code is temporalio/contrib/aws/lambda_worker/otel.py (apply_defaults). I have a fix ready that installs the provider once per process and reuses it on later calls (and shuts an unused provider down when another provider already owns the global slot, e.g. ADOT auto-instrumentation), with tests. Design notes I would like your view on: the first call wins (endpoint and service name are fixed for the process, which matches how they are configured today), and a foreign global provider still triggers OpenTelemetry's own "Overriding" warning on each call.

Lingua principale
Python
Stelle
1.2k
Fork
285
Merge medio
2g 1h
PR unite (30g)
81

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di temporalio/sdk-python

Tutte le issue di temporalio/sdk-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.