Add feature to enable connection diversity to a single upstream

Title: Add feature to enable connection diversity to a single upstream

Description: We would like the ability to tune the HTTP connection pool logic to create more connections to certain upstreams.

Typically, each upstream would have only 1-2 connections associated with them. This can be problematic when the endpoint represents multiple true backends behind it.

For example, say I am hitting storage.googleapis.com. Using a single connection for all my requests will ultimately be slower. This is explicitly recommended/used by Google APIs in a few places:

Developers on Google’s cloud platform will now also be able to upload their files faster thanks to Gsutil 3.4, which now uploads large objects in parallel. This update automatically uploads larger files over multiple connections to increase TCP throughput [1]

As we found out, if those file names are sequential then you could end up in a situation where multiple connections get squashed down to a single upload thread (thus hurting performance) [2]

Another case is when we are hitting a non-HTTP load balancer. https://github.com/istio/istio/issues/46959 is one such example.

Another example is Istio multi-network. In this model, we have an EDS config looking like this:

priority0:
  - local-pod-0
  - local-pod-1
  - local-pod-2
priority1:
  - single-remote-gateway-ip
    weight: 7 # represents workloads behind this one IP

In this case, we want to introduce connection diversity only to single-remote-gateway-ip. Without this, we will likely open up only 1-2 connections and hit only 1-2 backends on the remote side, resulting in uneven load. If we were to open up more connections, we have a much more probabilistic chance of having a more even load.