envoyproxy/envoy

xDS ACK/NACK are not sufficient for data plane health monitoring

Open

#35,675 opened on Aug 12, 2024

View on GitHub
 (6 comments) (0 reactions) (0 assignees)C++ (5,373 forks)batch import
area/xdsbughelp wanted

Repository metrics

Stars
 (27,997 stars)
PR merge metrics
 (Avg merge 8d) (378 merged PRs in 30d)

Description

Current xDS protocol feedback semantics is too limiting to validate the configuration is effective on the data plane:

  • ACK is per group of resources, and application can be partial where only a subset of resources is applied;
  • NACK is not useful since the only possible response is to retry, and a retry loop is harmful;
  • NACK is required to handle external references (sockets held by another process, files synced by another controller)
  • ACK needs to be synchronous, which is not feasible for complex configuration, e.g. a Wasm filter that needs to be fetched, compiled, and loaded on all workers;
  • NACK needs to be synchronous, which is not feasible for inter-dependent resources, e.g.:
    • a route config that is invalid only on some listeners
    • a mismatched public and private key when loaded via SDS

It seems the flaw is that the inline ACK/NACK in xDS protocol is not expressive enough to cover all the possible error states. The only value of current ACK/NACK is denying "syntactically" incorrect responses, which can be invalidated immediately but are quite rare. I believe a better architecture is needed, as mentioned in https://github.com/envoyproxy/envoy/issues/11396. We do not need the full complexity of reverse subscriptions, but the key is that the status is sent asynchronously, per resource, and independently from xDS requests.

Contributor guide