Metrics export runs on the application's tokio runtime, leaking internal telemetry (and hanging shutdown)

Open
  • Is something wrong?
#141 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
rust
Domain
observability-sre

Research direction

Start in spawn_runtime_and_exporters() and logfire/src/logfire.rs, focusing on meter_provider_builder.build() and the export runtime's handle.enter(). Run logfire/tests/test_http_sink.rs::test_http_metrics_export, then compare metrics-enabled export records and shutdown behavior under the tokio test configurations described. Done means metrics export no longer leaks internal HTTP-client telemetry and shutdown does not hang, including consideration of additional_readers.

Written by the indexing model from the issue text.

Description

Two symptoms, one root cause. Both need metrics export to Logfire enabled (.with_metrics(Some(MetricsOptions::default())) with send_to_logfire), and both require logfire::configure() to be called from inside a tokio runtime — i.e. any #[tokio::main] application.

Root cause

spawn_runtime_and_exporters() creates a dedicated, telemetry-suppressed export runtime and builds the PeriodicReader under handle.enter(), exactly as it does for the span/log batch processors. But unlike those processors, the async-runtime PeriodicReader does not spawn its worker task in build() — it stashes a closure and spawns it lazily from MetricReader::register_pipeline():

  • opentelemetry_sdk 0.32.1, src/metrics/periodic_reader_with_async_runtime.rs: build() stores ProducerOrWorker::Worker(...); register_pipeline() calls worker(self) → runtime.spawn(...) → tokio::spawn on whatever runtime is ambient at that moment.

register_pipeline() runs when the meter provider is built — meter_provider_builder.build() in Logfire::configure() (logfire/src/logfire.rs), on the caller's thread, outside the export runtime's handle.enter(). So the metrics worker task ends up on the application's runtime.

Symptom 1: internal HTTP-client telemetry is exported to Logfire

Because the worker isn't on the suppressed export runtime, the metrics export's own HTTP traffic generates telemetry that the SDK captures and ships.

Measured with a mock server, global (non-.local()) logfire, one real info! call and one counter increment: 15 of 17 exported log records were hyper-util / reqwest / mio internals — "checkout waiting for idle connection", "starting new connection", "http1 handshake complete, spawning background dispatcher task", "registering event source with poller", and so on. Every one of them carried thread.name = "tokio-runtime-worker" (the app runtime) rather than logfire-export-runtime. With metrics disabled, the same test exports exactly 1 record.

So users with metrics enabled are paying for, and wading through, a stream of connection-pool noise from the SDK's own exporter.

Symptom 2: shutdown() hangs

PeriodicReader::shutdown_with_timeout() sends a Message::Shutdown to that worker task and then futures_executor::block_ons the reply. If the worker is on the application's current_thread runtime and shutdown() is called from that runtime, the thread is blocked and the message can never be processed — deadlock:

Logfire::shutdown
  SdkMeterProvider::shutdown → Pipelines::shutdown → PeriodicReader::shutdown_with_timeout  [blocked forever]

Reproduces under a plain #[tokio::test]; does not reproduce under flavor = "multi_thread" with spawn_blocking. logfire/tests/test_http_sink.rs::test_http_metrics_export currently works around it with spawn_blocking.

Candidate fix

Enter the export runtime's handle around meter_provider_builder.build() so register_pipeline spawns the worker on logfire's runtime. Locally this dropped the leaked records to zero. Open questions: how it should interact with user-supplied additional_readers (registered by the same build() call), and whether it fully resolves the hang for ShutdownGuard dropped at the end of an async main. Under investigation; a PR may follow.

Possibly also worth an upstream report — spawning in register_pipeline rather than build() makes the reader's runtime depend on an unrelated later call site.

🤖 Generated with Claude Code

https://claude.ai/code/session_015jD9BNGcCVUpYWhjYniV6a

Dominant language
Rust
Stars
78
Forks
13
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydantic/logfire-rust

All issues in pydantic/logfire-rust

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.