area/ext_authzhelp wantedquestion
仓库指标
- Star
- (27,997 star)
- PR 合并指标
- (平均合并 8天) (30 天内合并 378 个 PR)
描述
Description:
What issue is being seen? Describe what should be happening instead of the bug.
We use ExtAuthz type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz filter. Many routes are configured in envoy config and want all requests to first pass through ExtAuthz filter for authentication.
When there is a load on Envoy say more than 15 RPS, we observed that for couple of requests Envoy doesn't call the ExtAuthz filter and just returns 503. looks like we are missing some configuration.
Below is the trimmed version of our envoy config and logs.
Envoy Configuration
admin:
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
http_protocol_options:
accept_http_10: true
stat_prefix: ingress_http
stream_idle_timeout: 900s
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
log_format:
json_format:
"start_time": "%START_TIME%"
"request_id": "%REQ(X-REQUEST-ID)%"
"user_agent": "%REQ(USER-AGENT)%"
"method": "%REQ(:METHOD)%"
"path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
"protocol": "%PROTOCOL%"
"upstream_protocol": "%UPSTREAM_PROTOCOL%"
"request_authority": "%REQ(:AUTHORITY)%"
"status": "%RESPONSE_CODE%"
"start_time_to_last_byte_out_ms": "%DURATION%"
"upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
"response_flag": "%RESPONSE_FLAGS%"
"response_code_details": "%RESPONSE_CODE_DETAILS%"
"hostname": "%HOSTNAME%"
"upstream_cluster": "%UPSTREAM_CLUSTER%"
"original-host": "%REQ(X-Forwarded-Host)%"
http_filters:
- name: envoy.filters.http.lua
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(request_handle)
end
source_codes:
some.lua:
inline_string: |
function envoy_on_request(request_handle)
request_handle:headers():add("X-Forwarded-Host", request_handle:headers():get("host"))
local path_without_querystring = string.match(request_handle:headers():get(":path"), "([^?]+)")
request_handle:headers():add("X-Forwarded-Path", path_without_querystring)
end
- name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
clear_route_cache: true
transport_api_version: V3
include_peer_certificate: true
status_on_error:
code: "InternalServerError"
http_service:
server_uri:
uri: localhost:8080
cluster: ext_authz-http-service
timeout: 100s
path_prefix: /auth
authorization_response:
allowed_upstream_headers:
patterns:
- exact: "authorization"
- exact: "correlationid"
authorization_request:
allowed_headers:
patterns:
- exact: "authorization"
- exact: "host"
- exact: "x-request-id"
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: some_local_route
virtual_hosts:
- name: some_domains
domains: ["*"]
routes:
- match:
prefix: "/api"
route:
timeout: 900s
cluster: some-cluster
auto_host_rewrite: true
append_x_forwarded_host: true
- match:
prefix: "/healthz"
route:
prefix_rewrite: "/health"
timeout: 1s
cluster: ext_authz-http-service
auto_host_rewrite: true
append_x_forwarded_host: true
typed_per_filter_config:
envoy.filters.http.ext_authz:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
clusters:
- name: some-cluster
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: some-cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: some-service.com
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: some-service.com
- name: ext_authz-http-service
connect_timeout: 0.250s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: ext_authz-http-service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: localhost
port_value: 8080
Logs
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][pool] [source/common/conn_pool/conn_pool_base.cc:215] [Tags: "ConnectionId":"15140"] destroying stream: 0 remaining
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][pool] [source/common/conn_pool/conn_pool_base.cc:484] [Tags: "ConnectionId":"15140"] client disconnected, failure reason:
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][http] [source/common/http/conn_manager_impl.cc:1925] [Tags: "ConnectionId":"15406","StreamId":"13145634703429981886"] Codec completed encoding stream.
2023-11-30 18:16:33.185
2023-11-30 18:16:33.185 'server', 'envoy'
2023-11-30 18:16:33.185 'date', 'Thu, 30 Nov 2023 18:16:33 GMT'
2023-11-30 18:16:33.185 ':status', '500'
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][http] [source/common/http/conn_manager_impl.cc:1820] [Tags: "ConnectionId":"15406","StreamId":"13145634703429981886"] encoding headers via codec (end_stream=true):
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][http] [source/common/http/filter_manager.cc:996] [Tags: "ConnectionId":"15406","StreamId":"13145634703429981886"] Sending local reply with details ext_authz_error
2023-11-30 18:16:33.185
2023-11-30 18:16:33.185 'content-type', 'text/plain'
2023-11-30 18:16:33.185 'content-length', '95'
2023-11-30 18:16:33.185 ':status', '503'
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][http] [source/common/http/async_client_impl.cc:123] async http request response headers (end_stream=false):
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][router] [source/common/router/router.cc:1281] [Tags: "ConnectionId":"0","StreamId":"15018213574545538674"] upstream reset: reset reason: connection termination, transport failure reason:
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][client] [source/common/http/codec_client.cc:158] [Tags: "ConnectionId":"15140"] request reset
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][client] [source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"15140"] disconnect. resetting 1 pending requests
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][connection] [source/common/network/connection_impl.cc:250] [Tags: "ConnectionId":"15140"] closing socket: 0
2023-11-30 18:16:33.185 [2023-11-30 18:16:33.185][18][debug][connection] [source/common/network/connection_impl.cc:663] [Tags: "ConnectionId":"15140"] remote close (edited)
Failures are random.
We tried playing around with below setting but no success.
upstream_connection_options:
tcp_keepalive:
keepalive_interval: 10
keepalive_time: 1
max_requests_per_connection: 1
Happy to share more logs if required. Please help. thanks