envoyproxy/envoy
在 GitHub 查看Outlier detection not happening for upstream response timeout cases with grpc calls
Open
#22,395 创建于 2022年7月26日
area/grpcarea/outlier_detectionbughelp wantedinvestigate
仓库指标
- Star
- (27,997 star)
- PR 合并指标
- (平均合并 8天) (30 天内合并 378 个 PR)
描述
Description:
While verifying outlier_detection and failover configuration in a simulated failure scenario, observed logs without UPSTREAM HOST and grpc-status: unavailable.
Was expecting outlier detection to capture the (upstream_)response_timeout errors but these error were not considered.
Repro steps:
Setup:
EnvoyProxy as part Istio sidecar : "version": "ed148b62dfb0dc79adc8c8573ced4806883389c0/1.19.2-dev/Clean/RELEASE/BoringSSL"
Simulated Scenario:
- Svc-one calls Svc-Two.
- Configure route timeout
timeout: 5sto the http route in Svc-Two.
cluster: outbound|8002||svc-two.svc-two-namespace.svc.cluster.local
timeout: 5s
maxGrpcTimeout: 5s
retryPolicy:
hostSelectionRetryMaxAttempts: "5"
numRetries: 2
retriableStatusCodes:
- 503
retryHostPredicate:
- name: envoy.retry_host_predicates.previous_hosts
retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
- Using NetworkPolicy block traffic from Svc-one pods to Svc-two pods.
- This will result in upstream_response_timeout error on the Svc-One pods.
Logs:
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:54.739441Z debug envoy http [C440203][S374698705624913161] request end stream
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:54.763924Z debug envoy upstream transport socket match, socket tlsMode-disabled selected for host with address 10.*.*.43:8002
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:54.764002Z debug envoy upstream DNS refresh rate reset for svc-two-svc.svc-two-playground.svc.cluster.local, refresh rate 30000 ms
...
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742415Z debug envoy router [C440203][S374698705624913161] upstream timeout
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742471Z debug envoy pool cancelling pending stream
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742483Z debug envoy router [C440203][S374698705624913161] canceled pool request
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742502Z debug envoy http [C440203][S374698705624913161] Sending local reply with details upstream_response_timeout
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742547Z debug envoy http [C440203][S374698705624913161] encoding headers via codec (end_stream=true):
svc-one-deployment-7d8dcc748-4v7tp istio-proxy ':status', '200'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'content-type', 'application/grpc'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'grpc-status', '14'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'grpc-message', 'upstream request timeout'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'date', 'Thu, 21 Jul 2022 05:21:59 GMT'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'server', 'envoy'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742592Z debug envoy http2 [C440203] stream closed: 0
svc-one-deployment-7d8dcc748-4v7tp istio-proxy [2022-07-21T05:21:54.739Z] "POST /proto.SvcTwoGroup/Cre***ber HTTP/2" 200 UT 52 0 5003 - "-" "grpc-java-netty/1.39.0" "d60dbd1e-8f2b-4d04-9b1e-f0fadc7e5b9e" "svc-two-svc.svc-two-playground.fkcloud.in:8002" "-" "svc-two-svc.svc-two-playground.fkcloud.in:8002" 240.240.0.87:8002 Unavailable