Successive per_downstream_connection http local_ratelimit filters seem to reduce effective tokens, rate limit too early

Title: Successive per_downstream_connection http local_ratelimit filters seem to reduce effective tokens, rate limit too early

Description:

On Envoy 1.22.x and 1.21.x, when adding multiple local_ratelimit filters to either the listener or vhost config, or both, with per_downstream_connection=true set, the effective rate is lower than any of the defined token buckets alone.

The expected behavior is that each token bucket be distinct and honored. For example, if a client sends 1000 rps and each request encounters two per_downstream_connection=true local_ratelimit filters, one with 500 tokens per second (max 500) and one with 600 tokens per second (max 600), the expected number of ok requests should be around 500, with 500 receiving 429. However, in practice, the number of successful requests is closer to 200-300, which is unexpected.

Repro steps: Envoy 1.22.x and 1.21.x

1. Configure a listener, or route/vhost or combination thereof such that a request passes through multiple local_ratelimit filters with per_downstream_connection=true set.
2. Send increasing rates of requests.
3. Notice that rate limiting occurs at a much earlier and lower level than expected.

Example load test tool:

echo 'GET https://envoy/200' | vegeta attack -insecure -header 'Host: envoy200ok.com' -connections 1 -http2 -keepalive -max-body -1 -rate 1000/1s -duration 10s -max-workers 1 -workers 1

Config:

The following config should permit around 660-662 requests per second.

filters:

- 
    name: envoy.filters.http.local_ratelimit.per_conn_a
    typed_config:
        '@type': type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
        stat_prefix: per_conn_a
        token_bucket:
            max_tokens: 600
            tokens_per_fill: 60
            fill_interval: 0.100s
        filter_enabled:
            default_value:
                numerator: 100
            runtime_key: local_rate_limit_enabled_per_conn_a
        filter_enforced:
            default_value:
                numerator: 100
            runtime_key: local_rate_limit_enforced_per_conn_a
        local_rate_limit_per_downstream_connection: true
- 
    name: envoy.filters.http.local_ratelimit.per_conn_b
    typed_config:
        '@type': type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
        stat_prefix: per_conn_b
        token_bucket:
            max_tokens: 601
            tokens_per_fill: 61
            fill_interval: 0.100s
        filter_enabled:
            default_value:
                numerator: 100
            runtime_key: local_rate_limit_enabled_per_conn_b
        filter_enforced:
            default_value:
                numerator: 100
            runtime_key: local_rate_limit_enforced_per_conn_b
        local_rate_limit_per_downstream_connection: true

Demonstation: With the two local rate limits given above, and sending 1000 requests per second for 10 seconds on a single connection: Note how only 3272 requests are able to complete successfully, not the expected 6600. If one of either rate limits is commented out, rate limiting behaves as expected. If local_rate_limit_per_downstream_connection is removed, rate limiting behaves as expected. Vegeta output

"status_codes":{"200":3270,"429":6716},"errors":["429 Too Many Requests"]

Envoy stats

per_conn_a.http_local_rate_limit.enabled: 9988
per_conn_a.http_local_rate_limit.enforced: 6716
per_conn_a.http_local_rate_limit.ok: 3272
per_conn_a.http_local_rate_limit.rate_limited: 6716
per_conn_b.http_local_rate_limit.enabled: 3272
per_conn_b.http_local_rate_limit.enforced: 0
per_conn_b.http_local_rate_limit.ok: 3272
per_conn_b.http_local_rate_limit.rate_limited: 0

If envoy.filters.http.local_ratelimit.per_conn_b is commented out and the loadtest re-performed, the rate limiting behaves as expected: vegeta

"status_codes":{"200":6538,"429":3462},"errors":["429 Too Many Requests"]

envoy stats

main_thread.watchdog_miss: 0
per_conn_a.http_local_rate_limit.enabled: 10002
per_conn_a.http_local_rate_limit.enforced: 3462
per_conn_a.http_local_rate_limit.ok: 6540
per_conn_a.http_local_rate_limit.rate_limited: 3462
runtime.admin_overrides_active: 0

If you add another copy of the per_downstream_connection local rate limit (C with rate 603 per second) the effective rate drops further:

"status_codes":{"200":2180,"429":7820},"errors":["429 Too Many Requests"]

What I have tried:

• Using distinct names, runtime_keys, max token/per fill token values, fill intervals, runtime key names
• Tried with much smaller and larger values for max_tokens, tokens_per_fill, fill_interval. Seems to happen no matter what these are set to.

What I think is going on: It looks like, for each http_local_rate_limit token bucket configured, each request is taking that many tokens from the first bucket encountered. Then when requests reach the next bucket, they take a number of tokens, equal to the number of filters/buckets, from the first bucket encountered again. So if we have 3 rate limits with local_rate_limit_per_downstream_connection set, 3 tokens per request are being taken from the first bucket encountered. And so on.