prombench: Is Prombench reliable enough? fix or document ignorable baseline

NOTE: Prombench is implemented mostly in test-infra repo, but adding issue here for visibility, as we make decisions based on prombench in this repo.

Prombench is relatively unreliable. With the recent proposed updates to the dashboard that shows and the average percentages of the difference across core metrics it yields extra +/-~3% allocs/s, +/-~6% CPU use, +/-3% HTTP latency

The unreliability yields wasted time on chasing non-existing issues (e.g. https://github.com/prometheus/prometheus/issues/18022), or it masks existing issues.

What's funny, it tends to be the "PR" node that is mostly worse for some reasons (order of creation?)

Perhaps it's the effect of exact average calculations vs the YOLO "it looks kind of good to me" look on a graph, and it was always there, but in this issue I'd like to see what we can do to either:

• Understand and improve reliability (especially on CPU side)
• If we can't improve, document and improve dashboards to show the typical variance and when (how long to wait) e.g. make the numbers white if within acceptable ranges.

Some tips on what we see from profile are here and it suggests kernel page behavior with mmaps, but it's a long shot.

Ideas

A) To rule out accidental use of irrelevant commits, I propose we extend prombench start logic to always build both PRs from source by default: https://github.com/prometheus/test-infra/pull/940 B) Find current variance baseline and document, I can update dashboard PR for i C) Investigate deeper e.g.

• Load we put to both
• Run PMU (e.g. https://github.com/perfgo/perfgo) to understand what happens with the process that gets worse (e.g. on cache misses).

Evidences

I ran exact 1:1 Prometheus's via custom prombench scenario: https://github.com/prometheus/prometheus/pull/18036#issuecomment-3870073553

Both using quay.io/prometheus/prometheus:{{ .RELEASE }}

From what I looked so far, the load on both Prometheus-es are equal (queries, samples/s)

Yet the variance is clear