[Bug]: DD_TRACE_OTEL_ENABLED: OTel spans in sampling-rejected traces leave an unclosed empty-name ddtrace span (orphans; extra memory in FrankenPHP worker mode)

Offen
  • Stimmt etwas nicht?
#4,366 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
52/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
php
Bereich
backend, observability

Rechercherichtung

Start with src/DDTrace/OpenTelemetry/SpanBuilder.php, especially startSpan() around the sampling decision and the drop branch; then read the cited context handling in src/DDTrace/OpenTelemetry/Context.php. Reproduce the rejected sampling case using the linked standalone script and check whether dropped spans remain open. Done means rejected traces have no empty-name orphan spans or state persisting across worker requests.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Bug report

With DD_TRACE_OTEL_ENABLED=true and a sampling rule that rejects the trace (e.g. DD_TRACE_SAMPLING_RULES=[{"sample_rate":0.05}]), every OpenTelemetry-API span started in a rejected trace leaves behind an unclosed ddtrace span with an empty name and resource, whose parent isn't in the trace. We expect the same with Adaptive Sampling, which uses sampling rules. Kept traces and 100%-sampled traces are clean.

In FrankenPHP worker mode the leaked state survives the request:

  • Memory grows faster.
  • In some setups, one trace per worker accumulates hundreds of these spans for the worker's lifetime.

Source (tag 1.26.1):

  • src/DDTrace/OpenTelemetry/SpanBuilder.php#L190 and #L210: startSpan() opens a ddtrace span with \DDTrace\start_trace_span() before and independent of the sampling decision (shouldSample() at L200).
  • #L242-L246: when the decision is neither RECORD_AND_SAMPLE nor RECORD_ONLY, it returns Span::wrap($spanContext) before setting $span->resource = $this->spanName (L246). Nothing closes the ddtrace span opened above, so it stays open with an empty name.
  • src/DDTrace/OpenTelemetry/Context.php#L228: self::storage()->attach($currentContext); // TODO: Handle Detach. In the repro the OTel context storage still held a context between requests.
  • Rejected (priority −1) traces produce unsampled OTel parent contexts. The trace flags come from the ddtrace traceparent, Context.php#L186-L188. So the SDK's default ParentBased sampler drops each child span, which takes the drop branch above. The repro logged startSpan() returned OpenTelemetry\API\Trace\NonRecordingSpan, priority=-1.

Standalone reproduction: https://gist.github.com/Ardenexal/35129bc3bddd96386c6c063e635af496 (FrankenPHP worker script plus open-telemetry/sdk 1.14.0 from Packagist, no framework; bash run.sh):

  • An OTel hook() on an interface method starts and ends an internal span. The pre-hook does spanBuilder()->setParent(Context::getCurrent())->startSpan() plus Context::storage()->attach(); the post-hook does detach() plus end(). 3 hooked calls per request.
  • 8 workers.
  • Trace sink: ddapm-test-agent v1.66.0 sharing the network namespace.
  • 3,000 requests at ab -c 8 per variant, each in a fresh container.
Variant Traces One-root traces Empty-name spans Priorities RSS per 500 requests
5% rule, bridge on 3,000 156 2,844 (every rejected trace has one, as an orphan root) −1: 2,844, 2: 156 +8.1 MiB
100%, bridge on (control) 3,000 3,000 0 1: 3,000 +3.8 MiB
5% rule, DD_TRACE_OTEL_ENABLED=false (control) 3,000 3,000 0 −1: 2,845, 2: 155 +0.4 MiB

RSS returned to about 126–128 MiB after a worker restart in every variant. Literal result line:

RESULT 1-rules5-otel/final (after worker restart): traces=3000 spans=6312 one_root_traces=156 multi_root_traces=2844 traces_gt100_spans=0 empty_name_spans=2844 (kept=0 rejected=2844 as_root=2844) priorities={'-1': 2844, '2': 156}

Per-worker accumulation (partially reproduced). Add a hooked call that runs after frankenphp_handle_request() returns (as Symfony's terminate() does in worker mode), with a ddtrace-native span inside it. With those, 8 traces (one per worker) accumulated 431–531 spans each, 214–264 of them empty-name, over 60.6–77.0 s. That variant also lost 929 of 3,000 request traces. In our app the same symptom reached 1,845–2,367 empty-name spans per worker trace, with memory +25 MiB per 500 requests against +10 at 100%.

Notes:

  • The bridge grows RSS in worker mode even at 100% sampling (+3.8 against +0.4 MiB per 500 requests with it off). Rejection roughly doubles it.
  • OTEL_TRACES_SAMPLER=always_on makes the orphans disappear, but it isn't a clean workaround. In rejected traces the OTel spans then carry priority 1 while their root carries −1, which we suspect a real Agent treats as separately kept chunks. That's an inference, not verified.
  • Observed with the test agent only; worker mode on vs off was not compared.

Expected: a span the sampler drops must not leave an open ddtrace span: close or discard it, or don't open it before sampling. Rejected traces shouldn't carry empty-name orphans, and nothing should persist across requests in worker mode.

PHP version

PHP 8.3.36 (cli) (built: Oct 21 2026 16:00:24) (ZTS). FrankenPHP v1.13.2, worker mode.

Tracer or profiler version

ddtrace 1.26.1 (ZTS build). The repro doesn't load the profiler.

Installed extensions

ddtrace (emulated opentelemetry) plus the official image's defaults. open-telemetry/sdk 1.14.0, open-telemetry/api 1.9.0, open-telemetry/context 1.5.0.

Output of phpinfo()

Not attached.

Upgrading from

New installation; previously the opentelemetry PECL extension without ddtrace.

Vorherrschende Sprache
PHP
Sterne
558
Forks
192
Ø Merge
2 T. 3 Std.
Gemergte PRs (30 T.)
104

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus DataDog/dd-trace-php

Alle Issues in DataDog/dd-trace-php

Ähnliche Issues

Weitere Issues zu PHP

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.