[Bug]: Laravel endpoint collection fails with EMSGSIZE when the app has ~3,000 routes and is retried on every request

オープン
  • 何か問題がありますか？
#4,371 コメント 5 件 リアクション 1 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
laravel, linux, php
領域
backend, distributed-systems

調査の方向性

Start with the LaravelIntegration hook around Router::findRoute and the add_endpoint/flush_endpoints calls described in the issue, then trace how libdd-ipc sets max_message_size() and how blocking::enqueue_actions sends EnqueueActions requests. Reproduce with a route batch large enough to exceed the effective SOCK_SEQPACKET limit. Done means endpoint collection succeeds for large route sets without retrying the full collection on each request.

索引モデルが issue の本文から書いたものです。

説明

Bug report

Summary

In the Laravel integration, endpoint collection (add_endpoint + flush_endpoints) sends every route to the sidecar in a single IPC message. With enough routes the message exceeds the AF_UNIX SOCK_SEQPACKET limit and sendmsg fails with EMSGSIZE. The error is swallowed, are_endpoints_collected() never becomes true, and the whole collection is repeated on every request.

Environment

  • dd-trace-php 1.24.2, 1.25.1, 1.26.0, 1.26.1 and 1.26.2 (same behavior in all of them)
  • PHP 8.3 (NTS), php-fpm + Apache, Linux, Laravel 11 with route:cache, AppSec and profiling installed (AppSec enabled)
  • ~3,000 routes (php artisan route:list --json | jq length)

What happens

LaravelIntegration (Router::findRoute hook) does, on each request while !\DDTrace\are_endpoints_collected():

foreach ($router->getRoutes() as $route) { \DDTrace\add_endpoint(...); }
\DDTrace\flush_endpoints();

strace -f -e trace=sendmsg on a php-fpm worker shows the batch going out as one message that fails:

sendmsg(8, {msg_iov=[{iov_base="...", iov_len=427136}], ...}, 0) = -1 EMSGSIZE (Message too long)

The batch is ~53 bytes per route plus the strings (3,037 routes ≈ 427 KB). The socket buffer is requested as 4 MiB, but the kernel caps SOCK_SEQPACKET messages at about 2 x net.core.wmem_max (212,992 by default, so ~425,984). net.core.wmem_max is not tunable per container (the file does not exist in the container netns), so containers cannot raise the limit.

With the batch just below the limit (e.g. dropping ~20 routes), collection succeeds and are_endpoints_collected() becomes true, so the problem appears suddenly when an app crosses ~3,000 routes.

Impact

  • The collection is rebuilt on every request (CompiledRouteCollection::getRoutes() -> ~3,000 newRoute), about +0.8 vCPU per busy worker pool in our measurements.
  • The sidecar transport is closed and reconnected on each request (The sidecar transport is closed. Reconnecting...) and AppSec logs PHP Warning: [ddappsec] Error in message exchange for command request_shutdown: dd_helper_fatal on every request.
  • In our production deployment, tasks that crossed the limit started writing to disk at ~100 MB/s about 1-2 minutes after receiving traffic and filled their 20 GiB ephemeral storage in 2-3 minutes. We could not conclusively identify the writing process (the sidecar log files are regular files named pipe:[inode] created in the working directory when error_log=/dev/stderr), so we are not claiming the sidecar is the writer; the strong correlation is with this failure loop. The same image on a staging with ~200x less traffic did not fill the disk.

Code reading (master / main)

  • libdd-ipc requests a 4 MiB socket buffer (SOCKET_BUFFER_SIZE) and uses it as max_message_size(), but the effective Linux limit is lower.
  • blocking::enqueue_actions sends all actions in one EnqueueActions request and ignores the result (let _ = transport.with_retry(...)).
  • No config option (DD_API_SECURITY_..., appsec, telemetry) disables endpoint collection in the Laravel integration; DD_INSTRUMENTATION_TELEMETRY_ENABLED=false, DD_APPSEC_ENABLED=false, DD_TRACE_SIDECAR_CONNECTION_MODE=thread and DD_TRACE_SIDECAR_TRACE_SENDER=false individually do not avoid the EMSGSIZE.

Expected behavior

Endpoint collection should not fail or loop when an application has many routes. Either split the batch into several messages below the effective limit, or at least stop retrying on every request when the send fails, and expose a setting to disable the collection.

Workaround

A global middleware that calls add_endpoint and flushes every ~150 KB while !are_endpoints_collected() avoids the error (3 messages of ~146 KB for 3,037 routes; are_endpoints_collected() becomes true after the first message and the agent receives all 3,037 endpoints in app-endpoints).

Related

This is the same pattern as #3760 (WordPress integration collecting endpoints on every request until are_endpoints_collected() is true), which was resolved in 1.17.1 by disabling the collection for WordPress.

主要言語
PHP
スター
558
フォーク
192
平均マージ
2日 11時間
マージ済み PR（30日）
91

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

DataDog/dd-trace-php のほかの issue

DataDog/dd-trace-php の issue をすべて見る

似ている issue

PHP の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。