feat: Avoid decay of our micro-benchmarks; test them on CI
#14.926 aberto em 18 de set. de 2024
Métricas do repositório
- Stars
- (64.042 stars)
- Métricas de merge de PR
- (Mesclagem média 11d 5h) (118 fundiu PRs em 30d)
Description
Proposal
As per https://github.com/prometheus/prometheus/pull/14925, our code tends to change, but microbenchmarks (func BenchmarkXYZ(b *testing.B)) decay. They always build, but the they decay "semantically" by essentially have a broken flows (timeouts, deadlocks), they code they benchmark is suddenly wrongly configured (e.g. so you don't benchmark success case anymore but failures/incorrect execution).
Microbenchmarks decay because (currently) we don't require to re-run the benchmarks on every change of the related code.
-
For this reason in the past I wrote for my "Efficient Go" book a quick
testorbenchstructure that allows running benchmarks "as a test with a N == 1" just to test runnability.testorbenchalso allows doing more tests on!tb.IsBenchmark(), as you don't want to do detailed asserts during benchmark time as it's waste of time (success criteria should be deterministic) and you want to measure execution not tests. You can see example use here. -
One alternative would be to have some
go test -run ^$ -bench .* -benchtime 1xjust to at least see if things are runnable on CI. -
Finally, if we get the nightly microbenchmark compared to e.g. day before build setup, that would keep the whole pipeline fresh perhaps and give us meaningful data. Maybe that's a solution too (: See how Vitess is doing this.
Acceptance Criteria
- Our micro-benchmark code is not decaying e.g. it's semantics and flow is updated when the related code is updated.
cc @bboreham @cstyan
Honestly (3) would be nice to do!