bencheeorg/benchee

Provide "meta" statistics

Open

#56 opened on Dec 18, 2016

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Elixir (1,509 stars) (69 forks)batch import
enhancementgood first issuehelp wanted

Description

For different use cases, like PragTob/benchee_html#10 it'd be great to have statistics about statistics - what I'd call "meta statistics" - although there's probably some better real statistics name for this :)

What should be in there (that I know of so far):

  • job size (how many jobs are in there)
  • minimum of run times over all jobs
  • maximum of run times over all jobs

This should be added as a new key to the benchmarking suite (:meta_statistics) - it could be added within the statistics module but might be better off as a separate MetaStatistics step that is computed after general statistics have been computed.

Contributor guide

Tech stack
None
Domain
data
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Elixirfamiliarity with benchee codebasebasic statistics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the existing statistics module (likely lib/benchee/statistics.ex) to understand how current statistics are computed. Then implement a new step or add a new key :meta statistics to the benchmarking suite that calculates job size, minimum, and maximum run times across all jobs. Consider whether to integrate into the Statistics module or create a separate MetaStatistics step. Look at the issue comments for additional user requirements or constraints.