Simpler listener configuration for multiple domains with TLS support

Title: Simpler listener configuration for multiple domains with TLS support

Cross-post from envoy-users group where there was little response.

Description: tl;dr What is the simplest way to provide a mapping of domains to certificates and avoid generating repeated (one that can easily be templated by the domain+cert combination) filter chain configurations in the Envoy listener config?

Details:

The example for configuring multiple domains suggests configuring multiple filter chains. For example, using the examples in this document for two custom domains we have something like in our listener configurations:

filter_chains:
- filter_chain_match:
    server_names: ["foo.example.com"]
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/path/foo/cert.pem" }
          private_key: { filename: "/path/foo/key.pem" }
  # other configuration
  ...
- filter_chain_match:
    server_names: "bar.example.com"
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/path/bar/cert.pem" }
          private_key: { filename: "/path/bar/key.pem" }
  # other configuration (repeat of what we have for foo.example.com
  ...
# repeat for every other domain

When configuring listeners for many domains (in the order of thousands) this can become quite cumbersome as there is one entry per domain name with repeated associated configuration on other options and yields a very large configuration file (in the order of MBs). In addition, anytime a domain name gets added or removed, the configuration has to be regenerated (for LDS). Separately, use of CommonTlsContext means reloading the certificate needs an Envoy hot-restart.

The Envoy documentation of Certificate Management says:

Secret Discovery Service referenced certificates. By using SDS, certificates can either be referenced as files (reloading the certs when the parent directory is moved) or through an external SDS server that can push new certificates.

We explored the use of SDS file as documented in this article:

 common_tls_context:
     alpn_protocols: ["h2", "http/1.1"]
     tls_certificate_sds_secret_configs:
         sds_config:
             path: /etc/envoy/sds.yaml

But, SDS yaml files can contain only a single entry, and attempting to configure multiple domain mapping fails with the following message:

[2021-03-16 23:40:35.661][909395][warning][config] [source/common/config/filesystem_subscription_impl.cc:43] Filesystem config update rejected: Unexpected SDS secrets length: 251

As a result, using SDS with the file-based approach means we need to generate additional files, one per domain, to support dynamic refresh of certificates on top of the filter chain listener configurations described above.

Question:

What is the better alternative for configuring listeners for multiple domains along with TLS cert configuration? Ideally, we avoid repeating filter chain configurations and generating multiple files for certificates.

Relevant feature in HAProxy:

We have previously used HAProxy for similar configurations where a common “listener” configuration can be provided for multiple domains with the per domain certificate lookup dynamically configured via an external table (crt-list option). This yields a much smaller configuration file, in the order of KBs when everything is combined.

ECDS mentioned in user group:

There was some mention of ECDS in the user-group thread, but I was unable to find an example. Reading what little I was able to find about it looks promising. Do we have an idea what the example would look like for the dummy configuration with foo.example.com and bar.example.com I have above? A link to a good reference would also be much appreciated.

Thanks in advance for your help.