envoyproxy/envoy

Should a cluster (resource) asked by ondemand XDS be marked as "suscribed" after discover is completed?

Open

#25.924 aperta il 5 mar 2023

Vedi su GitHub
 (10 commenti) (1 reazione) (0 assegnatari)C++ (5373 fork)batch import
help wantedno stalebotquestion

Metriche repository

Star
 (27.997 star)
Metriche merge PR
 (Merge medio 8g) (378 PR mergiate in 30 g)

Descrizione

I'm using Envoy's on demand discovery plugin, together with an XDS server using the DELTA_GRPC protocol (configs below). In our setting, resources aren't pre-allocated but are created when envoy reaches out to XDS asking for information about a cluster it just got a request to. After a while, if the cluster doesn't get any traffic, the backend service is terminated, and the XDS communicates it nicely to Envoy (which also logs the removal of the cluster, and the cluster no longer appears in the envoy stats). This all works great, but after a restart of the XDS server, Envoy issues another request for the deleted cluster, causing a new resource to be created. Here's the full sequence, followed by some research we did as to why:

  1. A request comes in to Envoy, for cluster it doesn't know about yet.
  2. Envoy asked XDS for information about the cluster.
  3. XDS creates a service and responds to Envoy.
  4. Envoy routes the request succesfuly. ... some time passes ..
  5. The service is deleted, and XDS notifies Envoy of it.
  6. Envoy logs the cluster removal here and the cluster is no longer removed in envoy stats.
[2023-03-05 11:40:51.471][33752139][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 0 cluster(s), remove 1 cluster(s)

... more time passes ... 7) The XDS service is restarted, and Envoy re-establishes the connection. Part of the request being sent includes information for the cluster that was removed, causing it to be re-created again:

[2023-03-05 11:41:41.117][33752139][debug][config] [./source/common/config/grpc_stream.h:62] Establishing new gRPC bidi stream to xds_cluster for rpc DeltaClusters(stream .envoy.service.discover
y.v3.DeltaDiscoveryRequest) returns (stream .envoy.service.discovery.v3.DeltaDiscoveryResponse);
..
[2023-03-05 11:41:41.177][33752139][info][upstream] [source/common/upstream/cds_api_helper.cc:32] odcds: add 1 cluster(s), remove 0 cluster(s)

Looking at Envoy's code - it seems that when a cluster that was explicitly requested is removed, it is not really removed but is marked as setAsWaitingForServer . See here . This causes the same resource to be asked again upon reconnection - see here . The comment from the code reads:

  // So, leave the version map entry present but blank if we are still interested in the resource.
  // It will be left out of initial_resource_versions messages, but will remind us to explicitly
  // tell the server "I'm cancelling my subscription" when we lose interest. In case of resources
  // received as a part of the wildcard subscription or resources we already lost interest in, we
  // just drop them.

Which means that the code expects someone to indicate that the cluster is no longer of interest. Who should that someone be? Should it be the ondemand plugin that started the discovery (leaving the cluster lifecycle to be managed by XDS or envoy)?

XDS config:

dynamic_resources:
  cds_config:
    resource_api_version: V3
    api_config_source:
      api_type: DELTA_GRPC
      transport_api_version: V3
      grpc_services:
        envoy_grpc:
          cluster_name: xds_cluster

on demand config:

- name: envoy.filters.http.on_demand
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.on_demand.v3.OnDemand
    odcds:
      source:
          resource_api_version: V3
          api_config_source:
              api_type: DELTA_GRPC
              transport_api_version: V3
              grpc_services:
                envoy_grpc:
                  cluster_name: xds_cluster

Guida contributor