Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: Laravel endpoint collection fails with EMSGSIZE when the app has ~3,000 routes and is retried on every request

Aperta
  • Qualcosa non va?
#4,371 5 commenti 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
laravel, linux, php
Ambito
backend, distributed-systems

Direzione di ricerca

Start with the LaravelIntegration hook around Router::findRoute and the add_endpoint/flush_endpoints calls described in the issue, then trace how libdd-ipc sets max_message_size() and how blocking::enqueue_actions sends EnqueueActions requests. Reproduce with a route batch large enough to exceed the effective SOCK_SEQPACKET limit. Done means endpoint collection succeeds for large route sets without retrying the full collection on each request.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug report

Summary

In the Laravel integration, endpoint collection (add_endpoint + flush_endpoints) sends every route to the sidecar in a single IPC message. With enough routes the message exceeds the AF_UNIX SOCK_SEQPACKET limit and sendmsg fails with EMSGSIZE. The error is swallowed, are_endpoints_collected() never becomes true, and the whole collection is repeated on every request.

Environment

  • dd-trace-php 1.24.2, 1.25.1, 1.26.0, 1.26.1 and 1.26.2 (same behavior in all of them)
  • PHP 8.3 (NTS), php-fpm + Apache, Linux, Laravel 11 with route:cache, AppSec and profiling installed (AppSec enabled)
  • ~3,000 routes (php artisan route:list --json | jq length)

What happens

LaravelIntegration (Router::findRoute hook) does, on each request while !\DDTrace\are_endpoints_collected():

foreach ($router->getRoutes() as $route) { \DDTrace\add_endpoint(...); }
\DDTrace\flush_endpoints();

strace -f -e trace=sendmsg on a php-fpm worker shows the batch going out as one message that fails:

sendmsg(8, {msg_iov=[{iov_base="...", iov_len=427136}], ...}, 0) = -1 EMSGSIZE (Message too long)

The batch is ~53 bytes per route plus the strings (3,037 routes ≈ 427 KB). The socket buffer is requested as 4 MiB, but the kernel caps SOCK_SEQPACKET messages at about 2 x net.core.wmem_max (212,992 by default, so ~425,984). net.core.wmem_max is not tunable per container (the file does not exist in the container netns), so containers cannot raise the limit.

With the batch just below the limit (e.g. dropping ~20 routes), collection succeeds and are_endpoints_collected() becomes true, so the problem appears suddenly when an app crosses ~3,000 routes.

Impact

  • The collection is rebuilt on every request (CompiledRouteCollection::getRoutes() -> ~3,000 newRoute), about +0.8 vCPU per busy worker pool in our measurements.
  • The sidecar transport is closed and reconnected on each request (The sidecar transport is closed. Reconnecting...) and AppSec logs PHP Warning: [ddappsec] Error in message exchange for command request_shutdown: dd_helper_fatal on every request.
  • In our production deployment, tasks that crossed the limit started writing to disk at ~100 MB/s about 1-2 minutes after receiving traffic and filled their 20 GiB ephemeral storage in 2-3 minutes. We could not conclusively identify the writing process (the sidecar log files are regular files named pipe:[inode] created in the working directory when error_log=/dev/stderr), so we are not claiming the sidecar is the writer; the strong correlation is with this failure loop. The same image on a staging with ~200x less traffic did not fill the disk.

Code reading (master / main)

  • libdd-ipc requests a 4 MiB socket buffer (SOCKET_BUFFER_SIZE) and uses it as max_message_size(), but the effective Linux limit is lower.
  • blocking::enqueue_actions sends all actions in one EnqueueActions request and ignores the result (let _ = transport.with_retry(...)).
  • No config option (DD_API_SECURITY_..., appsec, telemetry) disables endpoint collection in the Laravel integration; DD_INSTRUMENTATION_TELEMETRY_ENABLED=false, DD_APPSEC_ENABLED=false, DD_TRACE_SIDECAR_CONNECTION_MODE=thread and DD_TRACE_SIDECAR_TRACE_SENDER=false individually do not avoid the EMSGSIZE.

Expected behavior

Endpoint collection should not fail or loop when an application has many routes. Either split the batch into several messages below the effective limit, or at least stop retrying on every request when the send fails, and expose a setting to disable the collection.

Workaround

A global middleware that calls add_endpoint and flushes every ~150 KB while !are_endpoints_collected() avoids the error (3 messages of ~146 KB for 3,037 routes; are_endpoints_collected() becomes true after the first message and the agent receives all 3,037 endpoints in app-endpoints).

Related

This is the same pattern as #3760 (WordPress integration collecting endpoints on every request until are_endpoints_collected() is true), which was resolved in 1.17.1 by disabling the collection for WordPress.

Lingua principale
PHP
Stelle
558
Fork
192
Merge medio
2g 3h
PR unite (30g)
104

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di DataDog/dd-trace-php

Tutte le issue di DataDog/dd-trace-php

Issue simili

Altre issue su PHP

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.