envoyMetricsService Istio Histograms are Missing +Inf Upper Bound
#12.387 geöffnet am 30. Juli 2020
Repository-Metriken
- Stars
- (27.997 Stars)
- PR-Merge-Metriken
- (Durchschn. Merge 8T) (378 gemergte PRs in 30 T)
Beschreibung
Title: envoyMetricsService Istio Histograms are Missing +Inf Upper Bound
Description: This is a problem with istio/proxyv2
With a normal sidecar/Kiali/Prometheus setup the Istio histogram metrics (ex. istio_request_duration_milliseconds_bucket) include the +Inf bucket.
With the metrics received by the gRPC envoyMetricsService the +Inf bucket is missing. This is a major issue for the histogram_quantile Prometheus function.
The highest bucket must have an upper bound of +Inf. (Otherwise, NaN is returned.)
For now I'll just convert the highest upper bound to +Inf in my metrics service.
[optional Relevant Links:] https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#RemoteService
Expected behavior
It should include the +Inf bucket. Here's what I receive right now:
[histogram:<sample_count:3 sample_sum:2565 bucket:<cumulative_count:0 upper_bound:0.5 > bucket:<cumulative_count:0 upper_bound:1 > bucket:<cumulative_count:0 upper_bound:5 > bucket:<cumulative_count:0 upper_bound:10 > bucket:<cumulative_count:0 upper_bound:25 > bucket:<cumulative_count:0 upper_bound:50 > bucket:<cumulative_count:0 upper_bound:100 > bucket:<cumulative_count:0 upper_bound:250 > bucket:<cumulative_count:0 upper_bound:500 > bucket:<cumulative_count:3 upper_bound:1000 > bucket:<cumulative_count:3 upper_bound:2500 > bucket:<cumulative_count:3 upper_bound:5000 > bucket:<cumulative_count:3 upper_bound:10000 > bucket:<cumulative_count:3 upper_bound:30000 > bucket:<cumulative_count:3 upper_bound:60000 > bucket:<cumulative_count:3 upper_bound:300000 > bucket:<cumulative_count:3 upper_bound:600000 > bucket:<cumulative_count:3 upper_bound:1.8e+06 > bucket:<cumulative_count:3 upper_bound:3.6e+06 > > timestamp_ms:1595617404646 ]
Steps to reproduce the bug
Run a gRPC envoy metrics service with an Istio sidecar and print the metrics received.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
client version: 1.6.4
control plane version: 1.6.4
data plane version: 1.6.4 (12 proxies)
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-04-16T11:44:03Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9+IKS", GitCommit:"dbe4e8a5c169e0a6a8a9b7b183561fa61e161985", GitTreeState:"clean", BuildDate:"2020-07-16T01:30:57Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}