Starting envoy with one file for CDS clusters, and one for EDS endpoints, with 400 clusters gets stuck handling inotify events.

Title: Starting envoy with one file for CDS clusters, and one for EDS endpoints, with 400 clusters gets stuck handling inotify events.

Description: Create a file for CDS containing 400 clusters in the format

  - "@type": type.googleapis.com/envoy.api.v2.Cluster
    name: my-service
    connect_timeout: 30s
    type: EDS
    eds_cluster_config:
      eds_config:
        path: "/mnt/envoy/envoy/apiproxy/endpoints.yaml"

Then create one file (endpoints.yaml) which contains endpoints for all the clusters with the format:

  - "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
    cluster_name: my-service
    endpoints:
      - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: 10.3.33.2
                  port_value: 5349
          - endpoint:
              address:
                socket_address:
                  address: 10.3.33.3
                  port_value: 5349

In the envoy config just use:

dynamic_resources:
  cds_config:
    path: "/mnt/envoy/envoy/apiproxy/clusters.yaml"

In the clean envoy docker ubuntu image latest, it will crash with the error

2019-01-31 07:29:48.247][001502][critical][assert] [source/common/filesystem/inotify/watcher_impl.cc:27] assert failure: inotify_fd_ >= 0.
[2019-01-31 07:29:48.247][001502][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0x5de
[2019-01-31 07:29:48.247][001502][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<0> obj</lib/x86_64-linux-gnu/libc.so.6> (If unsymbolized, use tools
/stack_decode.py):

If u raise

fs.inotify.max_queued_events

to a big value then it will keep on receiving inotify events, but won't start in any reasonable time (I waited a few minutes). It seems like each subsequent cluster starting its watch triggers inotify events for all previous Clusters. (which would make it O(n^2).

Creating different directories for each clusters endpoints (you can actually just copy the endpoints.yaml file containing endpoints for 400 services), so that each cluster watches its own directory makes envoy startup in 10-30s.