Envoy Prefers Expired Certificate for Validation and Fails Under Certain Circumstances, Despite Listing the Correct Certificate Under the `certs/` Endpoint

Title: Envoy Prefers Expired Certificate for Validation and Fails Under Certain Circumstances, Despite Listing the Correct Certificate Under the certs/ Endpoint

Description:

Envoy is preferring an expired certificate for validation, which causes it to fail under certain circumstances (detailed below). Additionally, Envoy lists only the valid certificate in the certs/ endpoint, even though it uses the expired certificate for validation.

This issue arose from the analysis of a real-world outage we faced at the end of last year, where a certificate expired in the trust store, causing Envoy to fail health checks and connections.

A SSCCE has been created to reproduce the issue and is documented below.

The expected behavior would be that Envoy should either:

• Pick the correct certificate
• Fail with an error log on startup
• Or at least print a warning about the setup.

Envoy should also list the correct certificate in the certs/ endpoint.

Repro Steps:

To reproduce the issue where Envoy fails to pick up the correct certificate from the trust store, follow the steps outlined in the reproducer setup available here: dastrobu/envoy-duplicate-cert-fail.

The steps include setting up a simple Envoy configuration with a hello world server (listener) and a client (cluster) which performs a TCP health check on the server.

The certificates for the reproducer are fully self-signed and set up using the scripts in the certs/ folder. See certs/README.md for details.

The certificate chain is summarized as follows:

graph LR
    root[Root CA]
    valid[Valid Intermediate]
    expired[Expired Intermediate]
    server[Valid Server]
    root --> valid
    root --> expired
    valid --> server
    expired --> server

An important detail is that the valid and expired certificates share the same public key and subject.

So the server certificate can be validated by either the root CA (best practice) or any of the intermediate certificates (depending on the time you live in).

Ideally, Envoy would pick the right intermediate certificate from the trust store to validate the server certificate if both are available, depending on the expiration date.

Keeping two certificates with the same public key and subject in the trust store is not a best practice, but this is what happened in our case due to a lack of knowledge about certificate handling in multiple teams.

Adding intermediate certificates is also not a best practice, though it is common in some organizations, as there are services that do not serve the intermediate certificates in the chain, making it impossible to validate the server certificate by the root CA only.

All best practices aside, Envoy should still pick the right certificate from the trust store if there are two intermediate certificates with the same public key and subject, which only differ in the expiration date.

The observed behavior is that Envoy prefers the expired certificate over the valid one, depending on the order in the trust store.

If the expired certificate is put last in the trust store, Envoy fails unexpectedly. If the expired certificate is put first in the trust store, Envoy succeeds.

Note that curl has the same kind of ordering issue but behaves the other way around (see reproducer for details).

What makes this situation even more confusing is that Envoy lists only one certificate in the certs/ endpoint, even though there are two certificates in the trust store. Additionally, the certificate listed under the certs/ endpoint is the valid certificate, not the expired one in the "trust expired cert last" scenario, which ultimately leads to the failure.

Equally confusing is the "trust expired cert first" setup, where Envoy lists only the expired certificate in the certs/ but still succeeds in the health check.

I'd expect that the certs/ endpoint would either list both certificates from the trust store or list the one that is actually picked for validation.

Admin and Stats Output:

See reproducer.

Config:

See reproducer.

Logs:

See reproducer.

Call Stack:

See reproducer.