Add overload/overflow header(information) to responses triggered by cluster max-retries reached
#15,168 建立於 2021年2月24日
倉庫指標
- Star
- (27,997 star)
- PR 合併指標
- (平均合併 8天) (30 天內合併 378 個 PR)
描述
Title: Add x-envoy-overloaded header OR upstream_rq_retry_overflow related header/body to responses triggered by Cluster maxretries reached
Description: Currently, when Cluster maximum pending requests or Cluster maximum requests is configured by circuit breaker, and we have requests beyond the threshold. We will get the local response(reply) from Envoy containing the x-envoy-overloaded header AND the body information upstream connect error or disconnect/reset before headers. reset reason: overflow, for example:
< HTTP/1.1 503 Service Unavailable
< content-length: 81
< content-type: text/plain
< **x-envoy-overloaded: true**
< date: Fri, 03 Jul 2020 18:14:58 GMT
< server: istio-envoy
<
Connection #0 to host 10.107.212.149 left intact
**upstream connect error or disconnect/reset before headers. reset reason: overflow**
This information indicates that Envoy considers this to be an overloaded case.
But when I configure Cluster maximum active retries to a small value and make requests such that the maxRetries is exceeded, I see that envoyproxy simply checks the maxRetries, and when it's exceeded, envoy simply forwards the faulty response from the upstreams and doesn't provide information about the upstream_rq_retry_overflow is reached. For example, my traffic is client--> ingressGW --> two servers(1 httpbin +1 faulty server always returns 504 ) So after the maxRetries is exceeded, envoy doesn't retry and returns me normal 504 response below without indicating that maxRetries is exceeded.
< HTTP/1.1 504 Gateway Timeout
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< date: Tue, 23 Feb 2021 11:48:38 GMT
< content-length: 6
< x-envoy-upstream-service-time: 5
< server: istio-envoy
<
Error
Connection #0 to host 10.107.33.39 left intact
Desired behavior: As Envoy states "Envoy will set the x-envoy-overloaded header on the downstream response if a request was dropped due to upstream circuit breaking." https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-overloaded.
It makes sense to me that Envoy should at least process the responses by adding the x-envoy-overloaded header or specific retry-overflow-related information to it, instead of directly forward the response to the client. Or the question is since both Cluster maximum active retries and Cluster maximum (pending) requests belong to the circuit breaker overflow, why the handling differs as such?
Think about the case: when we have many upstream servers deployed and most of them are overloaded, so the Cluster maximum active retries is easily reached, so it makes sense to indicate in the response that max retry is overflowed (in this specific scenario, the retry_overflow might potentially due to many upstreams are overloaded).
Therefore, the desired behavior in my mind would be either:
- Envoy returns the local reply with x-envoy-overloaded header and retry-overflow related body as Envoy did when Cluster maximum pending requests or Cluster maximum requests exceeded.
- Or simpler way: Envoy adds the x-envoy-overloaded header to the responses from upstreams before forwarding them to the client.
[optional Relevant Links:] related to https://github.com/envoyproxy/envoy/issues/15106