Add validation when a gRPC service is configured on a HTTP/1 cluster
#11,439 opened on Jun 4, 2020
Repository metrics
- Stars
- (27,997 stars)
- PR merge metrics
- (Avg merge 8d) (378 merged PRs in 30d)
Description
When we were prototyping using the ext_authz filter, we configured it to use gRPC to a "self cluster", i.e. one that points at a socket on the local Envoy sidecar. This is perhaps non-standard, but the reason we did this is that we wanted the requests to run through our typical egress filter chain.
We forgot to set http2_protocol_options on this cluster, and were quite confused when our gRPC upstream service was returning OK status codes, and we were still seeing requests get denied by the filter. What happened was the filter was making HTTP/1 requests to the self cluster, and then they were upgraded to http2 before being sent to the gRPC upstream. The response then had the gRPC trailers stripped, so the filter failed the requests.
The clue to what was happening was that our egress logs were showing HTTP/1 rather than HTTP/2, everything else including gRPC trailers in the access log looked fine!
Sample log: ( the -/0 is the grpc-status header followed by grpc-status trailer)
[2020-02-27T00:27:45.898Z] "POST /envoy.service.auth.v2.Authorization/Check HTTP/1.1" 200 -/0 - 2979 11 57 57 "-" "797296e2-5515-428a-945f-3459265494fa" "multipass_safe--staging" "-" "-" "10.1.3.17:32508"
This is probably an obscure enough issue to be low priority, but I have 2 possible suggestions:
- validate that the filter can only be configured with an h2 capable cluster when using gRPC (although this raises questions about when the validation should happen)
- force gRPC client requests to h2 even when the cluster is not configured for h2