envoyproxy/envoy

Outlier detection not happening for upstream response timeout cases with grpc calls

Open

#22.395 geöffnet am 26. Juli 2022

Auf GitHub ansehen
 (4 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)C++ (5.373 Forks)batch import
area/grpcarea/outlier_detectionbughelp wantedinvestigate

Repository-Metriken

Stars
 (27.997 Stars)
PR-Merge-Metriken
 (Durchschn. Merge 8T) (378 gemergte PRs in 30 T)

Beschreibung

Description: While verifying outlier_detection and failover configuration in a simulated failure scenario, observed logs without UPSTREAM HOST and grpc-status: unavailable. Was expecting outlier detection to capture the (upstream_)response_timeout errors but these error were not considered.

Repro steps: Setup: EnvoyProxy as part Istio sidecar : "version": "ed148b62dfb0dc79adc8c8573ced4806883389c0/1.19.2-dev/Clean/RELEASE/BoringSSL"

Simulated Scenario:

  • Svc-one calls Svc-Two.
  • Configure route timeout timeout: 5s to the http route in Svc-Two.
        cluster: outbound|8002||svc-two.svc-two-namespace.svc.cluster.local
        timeout: 5s
        maxGrpcTimeout: 5s
        retryPolicy:
          hostSelectionRetryMaxAttempts: "5"
          numRetries: 2
          retriableStatusCodes:
          - 503
          retryHostPredicate:
          - name: envoy.retry_host_predicates.previous_hosts
          retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
  • Using NetworkPolicy block traffic from Svc-one pods to Svc-two pods.
  • This will result in upstream_response_timeout error on the Svc-One pods.

Logs:

svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:54.739441Z      debug   envoy http      [C440203][S374698705624913161] request end stream
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:54.763924Z      debug   envoy upstream  transport socket match, socket tlsMode-disabled selected for host with address 10.*.*.43:8002
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:54.764002Z      debug   envoy upstream  DNS refresh rate reset for svc-two-svc.svc-two-playground.svc.cluster.local, refresh rate 30000 ms
...
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742415Z      debug   envoy router    [C440203][S374698705624913161] upstream timeout
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742471Z      debug   envoy pool      cancelling pending stream
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742483Z      debug   envoy router    [C440203][S374698705624913161] canceled pool request
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742502Z      debug   envoy http      [C440203][S374698705624913161] Sending local reply with details upstream_response_timeout
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742547Z      debug   envoy http      [C440203][S374698705624913161] encoding headers via codec (end_stream=true):
svc-one-deployment-7d8dcc748-4v7tp istio-proxy ':status', '200'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'content-type', 'application/grpc'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'grpc-status', '14'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'grpc-message', 'upstream request timeout'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'date', 'Thu, 21 Jul 2022 05:21:59 GMT'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 'server', 'envoy'
svc-one-deployment-7d8dcc748-4v7tp istio-proxy
svc-one-deployment-7d8dcc748-4v7tp istio-proxy 2022-07-21T10:51:59.742592Z      debug   envoy http2     [C440203] stream closed: 0
svc-one-deployment-7d8dcc748-4v7tp istio-proxy [2022-07-21T05:21:54.739Z] "POST /proto.SvcTwoGroup/Cre***ber HTTP/2" 200 UT 52 0 5003 - "-" "grpc-java-netty/1.39.0" "d60dbd1e-8f2b-4d04-9b1e-f0fadc7e5b9e" "svc-two-svc.svc-two-playground.fkcloud.in:8002" "-" "svc-two-svc.svc-two-playground.fkcloud.in:8002" 240.240.0.87:8002 Unavailable

Contributor Guide