Improve outbound connection balancing to balance across virtual listeners

Title: Improve outbound connection balancing to balance across virtual listeners

Description: We have a usecase with a set of hosts where each host maintains a single pair of long lived TCP connections (one inbound and one outbound) to each other host. All traffic between any pair of hosts is multiplexed over this single pair of connections. We're seeing problems with outbound connections from a given host not being balanced across Envoy workers even when we configure exact connection balancing on the outbound side. For example, with a 10 host configuration (9 outbound connections per host) with 6 workers, and exact connection balancing configured, one of the hosts shows the following distribution of outbound connections across workers:

envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.165.145_27010"} 1 envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.169.99_27010"} 1 envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.172.109_27010"} 1 envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.176.127_27010"} 1 envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.176.177_27010"} 1 envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.179.17_27010"} 1 envoy_listener_worker_0_downstream_cx_total{listener_address="10.10.188.61_27010"} 1 envoy_listener_worker_4_downstream_cx_total{listener_address="10.10.160.89_27010"} 1 envoy_listener_worker_4_downstream_cx_total{listener_address="10.10.181.214_27010"} 1

As far as I can tell, the reason the connection balancing isn't working better in this case is that we have 9 distinct virtual listeners (one per destination IP) and the connection balancer is only balancing across workers within a given virtual listener. Since we only have a single outbound connection per virtual listener/destination there really isn't any useful balancing that can be done within that virtual listener. What would work better in our case (or it seems any case with a very small number of long lived connections per destination) would be to balance outbound connections across workers globally within the Envoy instance (not on a per listener basis).

To test this theory I hacked some changes to increment a global sequence number every time an outbound connection is handed off to the per destination listener, and then select the worker number for a given connection on a global basis based on sequenceNum % workerCount. Code is here. This is a really ugly hack and I'm not suggesting it's the right design, but it did seem to give me much better outbound connection balancing:

envoy_listener_worker_0_downstream_cx_active{listener_address="10.10.238.59_27010"} 1 envoy_listener_worker_0_downstream_cx_active{listener_address="10.10.243.169_27010"} 1 envoy_listener_worker_1_downstream_cx_active{listener_address="10.10.234.233_27010"} 1 envoy_listener_worker_1_downstream_cx_active{listener_address="10.10.247.215_27010"} 1 envoy_listener_worker_2_downstream_cx_active{listener_address="10.10.230.35_27010"} 1 envoy_listener_worker_3_downstream_cx_active{listener_address="10.10.228.246_27010"} 1 envoy_listener_worker_4_downstream_cx_active{listener_address="10.10.232.143_27010"} 1 envoy_listener_worker_5_downstream_cx_active{listener_address="10.10.233.95_27010"} 1 envoy_listener_worker_5_downstream_cx_active{listener_address="10.10.255.143_27010"} 1

Can we consider enhancing the outbound connection balancing in some way so that it balances with a more global scope and not just within a given virtual listener/destination?