Endpoints in multiple priorities are treated independently

Title: Endpoints in multiple priorities are treated independently

Description: This is mostly a design proposal, since I think it would take a significant rework of the design to make the expected behavior happen

Current behavior

Currently if you have an endpoint in multiple priorities, they are treated independently wrt stats and healthchecking at the very least, even if they are identical in every way.

example:

resources:
- "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
  cluster_name: fake.main.cluster
  endpoints:
  - priority: 0
    lb_endpoints:
    - endpoint:
        address:
          socket_address:
            address: 172.22.0.6
            port_value: 80
        health_check_config:
          port_value: 6666
  - priority: 1
    lb_endpoints:
    - endpoint:
        address:
          socket_address:
            address: 172.22.0.6
            port_value: 80
        health_check_config:
          port_value: 6666
  - priority: 2
    lb_endpoints:
    - endpoint:
        address:
          socket_address:
            address: 172.22.0.6
            port_value: 80
        health_check_config:
          port_value: 6666
    - endpoint:
        address:
          socket_address:
            address: 172.22.0.8
            port_value: 80
        health_check_config:
          port_value: 6666
  - priority: 3

With active healthchecking, this will triple healthcheck the endpoints. It will also have three separate stats entries. I haven't verified, but i suspect that healthcheck failures for one, will not apply to the others.

My specific use-case for doing this sort of config is to have simpler locality-aware routing. i.e. assign an endpoint to all localities that it belongs and let Envoy handle it when it gets unhealthy.

The other reason to do this is for something like #3327 or #3930, where you are moving endpoints around in order to do traffic shaping, you want those to really act as moves instead of add/removes.

Expected behavior

I would expect Endpoints to be separate from their lb assignment. What i mean by that is that their lb assignment can be moved, duplicated, etc. and the endpoint's healthcheck data and stats are carried with it.

Options

Solve healthchecks only

One option is to implement something like HAProxy's track option. This would allow one endpoint's health status to be tied to another's so you'd only need to healthcheck one, but then notify all the dependents on healthcheck status change.

Hash the endpoints

One option would be to hash the endpoints somehow and treat equivalently hashed endpoints as one. This is kind of what #3959 does, but that special cases only during addition. I am proposing that they are treated as one unit for other things like stats and healthchecking.

We'd have to deal with what happens when the configuration of one changes, but not the other. Presumably they'd no longer hash the same and it would be treated as a "change".

Split EDS

Right now EDS has kind of combined both load balancing configuration and endpoint configuration. One option would be to have a separate list of endpoints and then assign them to load balancing sets using some identifier. Something like:

# "new" EDS
eds_endpoints:
- id: deadb33f
   address:
       socket_address:
           address: 172.22.0.6
           port_value: 80
   health_check_config:
       port_value: 6666
- id: 8badf00d
   address:
       socket_address:
           address: 172.22.0.8
           port_value: 80
   health_check_config:
       port_value: 6666
----
# Current EDS (renamed to LBDS or something?)
resources:
- "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
  cluster_name: fake.main.cluster
  endpoints:
    - priority: 0
       lb_endpoints:
          - eds_endpoint: deadb33f
    - priority: 1
       lb_endpoints:
          - eds_endpoint: deadb33f
    - priority: 2
       lb_endpoints:
          - eds_endpoint: deadb33f
            eds_endpoint: 8badf00d

This is effectively the same as the previous option, but puts the burden of the hashing on the management server.

The one difference here is that we could have incremental xDS send "modify" messages to modify existing endpoint configuration.