Use histograms for latency related metrics

Report

There are currently two identified latency metrics that are incorrectly implemented as gauges

• keda_scaler_metrics_latency_seconds
• keda_internal_scale_loop_latency_seconds

These originate from https://github.com/kedacore/keda/pull/5174 (which was rebased and merged through https://github.com/kedacore/keda/pull/5687). They deprecated keda_scaler_metrics_latency and keda_internal_scale_loop_latency, which were also gauges, but at the same time lacked the _seconds suffix recommended as best practice for naming prometheus metrics. During that PR, I should have also changed it from a gauge to a histogram but I didn't realize this at the time.

There are other metrics reported by keda-operator that relate to some form of latency. They come from vendored dependencies or golang core, all use histograms. A few examples I found are here:

• controller_runtime_reconcile_time_seconds
• controller_runtime_webhook_latency_seconds
• workqueue_work_duration_seconds
• workqueue_queue_duration_seconds
• go_sched_latencies_seconds
• keda_internal_metricsservice_grpc_server_handling_seconds

Expected Behavior

The keda_scaler_metrics_latency_seconds and keda_internal_scale_loop_latency_seconds should be either histograms or keda-operator should include new metrics that mirror these but as histograms

Actual Behavior

The keda_scaler_metrics_latency_seconds and keda_internal_scale_loop_latency_seconds are gauges with low information value for latency

Steps to Reproduce the Problem

1. deploy keda
2. check metrics endpoint
3. wonder why

Logs from KEDA operator

example

Would you be open to contributing a fix?

yes

Anything else?

might be best to have dual-write mirrored metrics, just replace _latency_ with _duration_ and document cleanly

• keda_scaler_metrics_latency_seconds keep as gauge -> keda_scaler_metrics_duration_seconds new as histogram
• keda_internal_scale_loop_latency_seconds keep as gauge -> keda_internal_scale_loop_duration_seconds new as histogram