SDS file watching action not triggered

Title: SDS file watching action not triggered

Description: The SDS file watch action does not get triggered in my setup.

The issue appeared in a kubernetes pod.

The pod has a container that regularly fetches certificates from vault and stores them in a shared volume attached to my envoy container.

Despite new certificates/keys being fetched, the SDS secrets do not get reloaded.

To verify that the inotify MOVED_TO event gets triggered by the certificate fetching container, I installed inotifywait in the envoy container and watched the shared volume. Doing so I can confirm that MOVED_TO are triggered when new certificates get fetched. I can also confirm that the content of the files in the watched directory has changed.

I attached my config to the issue,

I look forward to your support.

Note: If there are privacy concerns, sanitize the data prior to sharing.

Config:

static_resources:
  secrets:
    - name: server_cert
      tls_certificate:
        certificate_chain:
          filename: /envoy/certs/public.crt
        private_key:
          filename: /envoy/certs/private.key
    - name: server_ca
      validation_context:
        trusted_ca:
          filename: /envoy/certs/ca.crt
    - name: zookeeper_cert
      tls_certificate:
        certificate_chain:
          filename: /envoy/certs/zookeeper_public.crt
        private_key:
          filename: /envoy/certs/zookeeper_private.key
    - name: zookeeper_ca
      validation_context:
        trusted_ca:
          filename: /envoy/certs/zookeeper_ca.crt
  clusters:
    - name: internal
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: internal
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 9093
    - name: client
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: client
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 9092
    - name: zookeeper
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: zookeeper
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: zookeeper-0.zookeeper-headless.zookeeper.svc.cluster.local
                      port_value: 3181
              - endpoint:
                  address:
                    socket_address:
                      address: zookeeper-1.zookeeper-headless.zookeeper.svc.cluster.local
                      port_value: 3181
              - endpoint:
                  address:
                    socket_address:
                      address: zookeeper-2.zookeeper-headless.zookeeper.svc.cluster.local
                      port_value: 3181
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          common_tls_context:
            tls_certificate_sds_secret_configs:
              - name: zookeeper_cert
            validation_context_sds_secret_config:
              name: zookeeper_ca
  listeners:
    - name: listener_internal
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 19093
      filter_chains:
        - filters:
            - name: envoy.filters.network.kafka_broker
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.kafka_broker.v3.KafkaBroker
                stat_prefix: kafka_broker
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: tcp
                cluster: internal
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificate_sds_secret_configs:
                  - name: server_cert
                validation_context_sds_secret_config:
                  name: server_ca
    - name: listener_client
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 19092
      filter_chains:
        - filters:
            - name: envoy.filters.network.kafka_broker
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.kafka_broker.v3.KafkaBroker
                stat_prefix: kafka_broker
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: tcp
                cluster: client
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificate_sds_secret_configs:
                  - name: server_cert
                validation_context_sds_secret_config:
                  name: server_ca
    - name: listener_zookeeper
      address:
        socket_address:
          address: 127.0.0.1
          port_value: 3181
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: tcp
                cluster: zookeeper

Version:

envoy  version: dcd329a2e95b54f754b17aceca3f72724294b502/1.22.0/Clean/RELEASE/BoringSSL