Use benchmark.js or similar, report standard deviation
#23 aperta il 30 set 2016
Metriche repository
- Star
- (3761 star)
- Metriche merge PR
- (Nessuna PR mergiata in 30 g)
Descrizione
Right now I'm taking the median of 251 runs, but I'm a bit unsatisfied with this because I have not confirmed that 251 is enough to eliminate unreasonable variance, nor does it tell me the standard deviation or any of those niceties. As pointed out in https://github.com/nolanlawson/optimize-js/issues/2#issuecomment-248308234, we should really have a more rigorous benchmark in place, especially as we start to potentially make changes to "improve" performance, which need to be proven to actually improve it.
I looked at benchmark.js but TBH couldn't figure out how to nicely integrate it with the current benchmark design; e.g. currently I request each script with a random query param to force the browser to re-parse and re-evaluate it, and the timings have to be inside the script itself, so it's very tricky to implement.