[Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation

Start in temporalio/contrib/aws/lambda_worker/otel.py at apply_defaults, then read the run_worker configuration path and its existing tests. The issue provides a minimal reproduction using _run_worker_internal; run it across repeated invocations and check whether exporter threads and gRPC channels keep accumulating. Done means repeated calls do not grow those resources, including when another provider already owns the global slot.

What are you really trying to do?

Run a Temporal worker on AWS Lambda with tracing, following the README "Observability" example: call otel.apply_defaults(config, OtelOptions()) inside the configure callback passed to run_worker.

Describe the bug

run_worker calls the user's configure callback on every Lambda invocation (its docstring says so), and a warm container keeps the same process alive. apply_defaults builds a new replay-safe TracerProvider with a new BatchSpanProcessor(OTLPSpanExporter) each time (one daemon export thread and one gRPC channel) and then calls set_tracer_provider(). OpenTelemetry only lets the global provider be set once, so from the second invocation on:

• OpenTelemetry logs Overriding of current TracerProvider is not allowed,
• the new provider is never installed and never used (OpenTelemetryPlugin takes the global one), and
• it is never shut down, so its export thread and channel stay alive for the life of the container.

Result: one extra OtelBatchSpanRecordProcessor thread and gRPC channel per warm invocation, forever.

Expected: the documented per-invocation configure + apply_defaults pattern should be safe to run repeatedly, i.e. no growth in threads or channels across warm invocations.

Minimal Reproduction

Drives the real _run_worker_internal handler with a fake connect and a fake worker, so it needs no Lambda and no collector:

import gc, logging, threading

from opentelemetry import trace

from temporalio.common import WorkerDeploymentVersion
from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, otel
from temporalio.contrib.aws.lambda_worker._run_worker import _run_worker_internal, _WorkerDeps
from temporalio.envconfig import ClientConfigProfile

logging.basicConfig(level=logging.WARNING, format="LOG %(levelname)s %(name)s: %(message)s")


class FakeWorker:
    async def run(self):
        return


async def fake_connect(**kwargs):
    return object()


def configure(config: LambdaWorkerConfig) -> None:  # the README example
    config.worker_config["task_queue"] = "tq"
    otel.apply_defaults(config, otel.OtelOptions())


deps = _WorkerDeps(
    connect=fake_connect,
    create_worker=lambda client, **kw: FakeWorker(),
    load_config=lambda: ClientConfigProfile(address="localhost:7233"),
    getenv=lambda k: None,
)
handler = _run_worker_internal(WorkerDeploymentVersion("svc", "v1"), configure, deps)

providers = []
for invocation in range(1, 6):
    handler({}, None)  # one Lambda invocation in a warm container
    gc.collect()
    providers.append(trace.get_tracer_provider())
    n = sum(1 for t in threading.enumerate() if t.name == "OtelBatchSpanRecordProcessor")
    print(f"after invocation {invocation}: OTLP span-export threads alive={n}, total threads={threading.active_count()}")

print("global tracer provider is the same object every time:", len({id(p) for p in providers}) == 1)

Output on main:

LOG WARNING opentelemetry.trace: Overriding of current TracerProvider is not allowed   (x4)
after invocation 1: OTLP span-export threads alive=1, total threads=2
after invocation 2: OTLP span-export threads alive=2, total threads=3
after invocation 3: OTLP span-export threads alive=3, total threads=4
after invocation 4: OTLP span-export threads alive=4, total threads=5
after invocation 5: OTLP span-export threads alive=5, total threads=6
global tracer provider is the same object every time: True

Environment/Versions

• OS and processor: macOS (Apple Silicon), Python 3.12.14
• Temporal Version: temporalio main at 2a7d321c (version 1.35.0), with the native bridge from the 1.34.0 wheel; only pure-Python code from main differs. Extras opentelemetry-exporter-otlp-proto-grpc and opentelemetry-sdk-extension-aws installed.
• Docker or Kubernetes or building Temporal from source? No. This is a simulated invocation loop; I have not run it on a real Lambda or against a real collector.

Additional context

The relevant code is temporalio/contrib/aws/lambda_worker/otel.py (apply_defaults). I have a fix ready that installs the provider once per process and reuses it on later calls (and shuts an unused provider down when another provider already owns the global slot, e.g. ADOT auto-instrumentation), with tests. Design notes I would like your view on: the first call wins (endpoint and service name are fixed for the process, which matches how they are configured today), and a foreign global provider still triggers OpenTelemetry's own "Overriding" warning on each call.