Inconsistencies in gRPC rq_5xx stats

I'm looking into how an application can leverage Envoy stats to measure an SLO in terms of the ratio of "successful" requests to "failed" requests. The service I'm currently looking at supports both HTTP and gRPC for the same logical endpoint.

The first stat I looked at was the downstream_rq_5xx one, thinking that it might also include requests with a gRPC status code that maps to an HTTP 5xx code. I found that's not the case, although there are some edge cases when Envoy sends a local reply.

For instance: when an upstream timeout occurs, the timeout_response_code (defaults to 504) gets charged to the stats, and then this is converted to a 200 response code in sendLocalReply. This means that it's counted in upstream_rq_5xx, but NOT in downstream_rq_5xx because the conn manager sees the 200.

To summarize, there are two seemingly strange inconsistencies:

1. a gRPC error returned by the upstream (e.g. DEADLINE_EXCEEDED) will not be counted in upstream_rq_5xx, but one returned by Envoy local reply (e.g. upstream timeout) will be
2. Envoy local reply errors for gRPC requests count toward upstream_rq_5xx but not downstream_rq_5xx

I imagine we can't just change the meaning of these stats, but ideally we'd decide that either we always map gRPC codes to http codes and charge them in both upstream and downstream stats, or we never do and emit separate gRPC codes stats.