google/benchmark

Add support for thresholds file

Open

#339 opened on Feb 2, 2017

View on GitHub
 (3 comments) (1 reaction) (0 assignees)C++ (7,968 stars) (1,539 forks)batch import
enhancementgood first issuehelp wanted

Description

Good morning all I've read as much as I could the doc and list of issues on github but could nt find a reply regarding this question. If I understand right this benchmark library offers nice way to register and benchmark different functions. But I did nt find a way for the library to :

  • compare the timing (real, cpu, ...) with given threshold values
  • reports which benchmark passed and which one failed if the time is higher than the threshold I have tested this following solution but there should be better :
  • run the benchmark and output to json
  • use for example the jq command line tool in order to test that each benched functions are within the desired threshold (benchresults.json contains the json example as given in the main .md doc) : jq -e '.benchmarks[] | select(.name == "BM_SetInsert/1024/1").real_time < 30000' ~/tmp/benchresults.json

This is working but I m wondering if these tests should be better done inside the library. For instance, the user could provide via the commandline a "threshold" file (--treshold_file or whatever) giving the thresholds (time limits) for each benched functions : for example, in csv format : "BM_SetInsert/1024/1", 30000 "BM_SetInsert/1024/8", 33000 "BM_SetInsert/1024/10", 32000

RFC time. Cheers W.

Contributor guide

Tech stack
cpp
Domain
performancetesting
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C++ programmingGoogle Benchmark library basicsCommand line parsing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate how the library currently handles command line arguments and JSON output in files like bench.cc or benchmark.cc. Look for existing parsing of flags and output format. The feature would require adding a new command line flag to specify a thresholds file, parsing CSV (or similar) into a map of benchmark names to thresholds, and during benchmark reporting comparing real time against thresholds and producing a pass/fail output. Consider using the existing options.h or commandlineflags.h for integration.