Observed change in the load balancing behavior in v1.28.0

Title: Observed change in the load balancing behavior in v1.28.0

Description:

We are currently on envoy version 1.26.1. Trying to upgrade to v1.28.0 We see a behavioural change in envoy lb routing. Have attached below our cluster config (added via CDS mgmt server).

Setup: Two upstream cluster endpoints E1 and E2, single envoy. We are providing load_balancing_weight to each endpoint.

• E1 is in same/local zone of envoy, so our CDS server allocated weight of 2
• E2 is in cross zone, so weight is 1.

Expected Behavior:

• We expect roughly 66% requests to go to E1, 34% to go to E2. -> This is working behaviour as of 1.26.1

Actual Behavior:

• With v1.28.0, We get 49% and 51% distribution to E1 and E2 respectively. The endpoint weights of 2:1 are not being honored.

Can we please know what has changed. Or something incorrect with our expected behaviour.

We tried to test this with below flag set to false as suggested in the changelog, but its not giving us the old behavior. envoy.reloadable_features.locality_routing_use_new_routing_logic

        envoy:
          reloadable_features:
            locality_routing_use_new_routing_logic: false

Config:

{
     "version_info": "1270240254121585355",
     "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "evenly_distributed_cluster",
      "type": "STATIC",
      "connect_timeout": "15s",
      "lb_policy": "LEAST_REQUEST",
      "health_checks": [
       {
        "timeout": "4s",
        "interval": "10s",
        "interval_jitter": "1s",
        "unhealthy_threshold": 6,
        "healthy_threshold": 1,
        "tcp_health_check": {},
        "event_log_path": "/dev/stdout",
        "always_log_health_check_failures": true
       }
      ],
      "circuit_breakers": {
       "thresholds": [
        {
         "max_connections": 5000,
         "max_pending_requests": 5000,
         "max_requests": 5000,
         "retry_budget": {
          "budget_percent": {
           "value": 25
          },
          "min_retry_concurrency": 10
         }
        },
        {
         "priority": "HIGH",
         "max_connections": 5000,
         "max_pending_requests": 5000,
         "max_requests": 5000,
         "retry_budget": {
          "budget_percent": {
           "value": 25
          },
          "min_retry_concurrency": 10
         }
        }
       ]
      },
      "dns_refresh_rate": "5s",
      "dns_lookup_family": "V4_ONLY",
      "outlier_detection": {
       "interval": "10s",
       "base_ejection_time": "30s",
       "max_ejection_percent": 10,
       "consecutive_gateway_failure": 5,
       "split_external_local_origin_errors": true,
       "consecutive_local_origin_failure": 5
      },
      "common_lb_config": {
       "healthy_panic_threshold": {
        "value": 50
       },
       "locality_weighted_lb_config": {},
       "ignore_new_hosts_until_first_hc": true
      },
      "load_assignment": {
       "cluster_name": "evenly_distributed_cluster",
       "endpoints": [
        {
         "locality": {
          "region": "use1",
          "zone": "az6",
          "sub_zone": "az6"
         },
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "172.28.0.3",
              "port_value": 80
             }
            }
           }
          }
         ],
         "load_balancing_weight": 2
        },
        {
         "locality": {
          "region": "cross-zone",
          "zone": "cross-zone",
          "sub_zone": "cross-zone"
         },
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "172.28.0.16",
              "port_value": 80
             }
            }
           }
          }
         ],
         "load_balancing_weight": 1
        }
       ],
       "policy": {
        "overprovisioning_factor": 1
       }
      },
      "track_cluster_stats": {
       "timeout_budgets": true,
       "request_response_sizes": true
      }
     },
    },