Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug]: Laravel endpoint collection fails with EMSGSIZE when the app has ~3,000 routes and is retried on every request

Đang mở
  • Có gì không ổn?
#4,371 5 bình luận 1 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
laravel, linux, php
Lĩnh vực
backend, distributed-systems

Hướng nghiên cứu

Start with the LaravelIntegration hook around Router::findRoute and the add_endpoint/flush_endpoints calls described in the issue, then trace how libdd-ipc sets max_message_size() and how blocking::enqueue_actions sends EnqueueActions requests. Reproduce with a route batch large enough to exceed the effective SOCK_SEQPACKET limit. Done means endpoint collection succeeds for large route sets without retrying the full collection on each request.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Bug report

Summary

In the Laravel integration, endpoint collection (add_endpoint + flush_endpoints) sends every route to the sidecar in a single IPC message. With enough routes the message exceeds the AF_UNIX SOCK_SEQPACKET limit and sendmsg fails with EMSGSIZE. The error is swallowed, are_endpoints_collected() never becomes true, and the whole collection is repeated on every request.

Environment

  • dd-trace-php 1.24.2, 1.25.1, 1.26.0, 1.26.1 and 1.26.2 (same behavior in all of them)
  • PHP 8.3 (NTS), php-fpm + Apache, Linux, Laravel 11 with route:cache, AppSec and profiling installed (AppSec enabled)
  • ~3,000 routes (php artisan route:list --json | jq length)

What happens

LaravelIntegration (Router::findRoute hook) does, on each request while !\DDTrace\are_endpoints_collected():

foreach ($router->getRoutes() as $route) { \DDTrace\add_endpoint(...); }
\DDTrace\flush_endpoints();

strace -f -e trace=sendmsg on a php-fpm worker shows the batch going out as one message that fails:

sendmsg(8, {msg_iov=[{iov_base="...", iov_len=427136}], ...}, 0) = -1 EMSGSIZE (Message too long)

The batch is ~53 bytes per route plus the strings (3,037 routes ≈ 427 KB). The socket buffer is requested as 4 MiB, but the kernel caps SOCK_SEQPACKET messages at about 2 x net.core.wmem_max (212,992 by default, so ~425,984). net.core.wmem_max is not tunable per container (the file does not exist in the container netns), so containers cannot raise the limit.

With the batch just below the limit (e.g. dropping ~20 routes), collection succeeds and are_endpoints_collected() becomes true, so the problem appears suddenly when an app crosses ~3,000 routes.

Impact

  • The collection is rebuilt on every request (CompiledRouteCollection::getRoutes() -> ~3,000 newRoute), about +0.8 vCPU per busy worker pool in our measurements.
  • The sidecar transport is closed and reconnected on each request (The sidecar transport is closed. Reconnecting...) and AppSec logs PHP Warning: [ddappsec] Error in message exchange for command request_shutdown: dd_helper_fatal on every request.
  • In our production deployment, tasks that crossed the limit started writing to disk at ~100 MB/s about 1-2 minutes after receiving traffic and filled their 20 GiB ephemeral storage in 2-3 minutes. We could not conclusively identify the writing process (the sidecar log files are regular files named pipe:[inode] created in the working directory when error_log=/dev/stderr), so we are not claiming the sidecar is the writer; the strong correlation is with this failure loop. The same image on a staging with ~200x less traffic did not fill the disk.

Code reading (master / main)

  • libdd-ipc requests a 4 MiB socket buffer (SOCKET_BUFFER_SIZE) and uses it as max_message_size(), but the effective Linux limit is lower.
  • blocking::enqueue_actions sends all actions in one EnqueueActions request and ignores the result (let _ = transport.with_retry(...)).
  • No config option (DD_API_SECURITY_..., appsec, telemetry) disables endpoint collection in the Laravel integration; DD_INSTRUMENTATION_TELEMETRY_ENABLED=false, DD_APPSEC_ENABLED=false, DD_TRACE_SIDECAR_CONNECTION_MODE=thread and DD_TRACE_SIDECAR_TRACE_SENDER=false individually do not avoid the EMSGSIZE.

Expected behavior

Endpoint collection should not fail or loop when an application has many routes. Either split the batch into several messages below the effective limit, or at least stop retrying on every request when the send fails, and expose a setting to disable the collection.

Workaround

A global middleware that calls add_endpoint and flushes every ~150 KB while !are_endpoints_collected() avoids the error (3 messages of ~146 KB for 3,037 routes; are_endpoints_collected() becomes true after the first message and the agent receives all 3,037 endpoints in app-endpoints).

Related

This is the same pattern as #3760 (WordPress integration collecting endpoints on every request until are_endpoints_collected() is true), which was resolved in 1.17.1 by disabling the collection for WordPress.

Ngôn ngữ chính
PHP
Star
558
Fork
192
Merge trung bình
2 ngày 3 giờ
Pull request đã merge (30 ngày)
104

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của DataDog/dd-trace-php

Tất cả issue của DataDog/dd-trace-php

Issue tương tự

Thêm issue về PHP

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.