Separating benchmarks with different complexity and benchmarks with just variants

Open
#190 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Documentation
Clarity
Needs clarification
Activity status
Stale
Tech stack
ruby
Domain
documentation, performance

Research direction

Start by reviewing the repository’s benchmark examples, including the linked Array#bsearch versus Array#find example, and compare the CRuby results with the referenced TruffleRuby benchmarks. Done should separate complexity-driven results from implementation-sensitive variants and add the proposed caveats, but the issue does not identify files or tests.

Written by the indexing model from the issue text.

Description

Hello there,
I think it would be worthwhile to separate the example in two categories:

  • Benchmarks which are faster due to the variants having different complexity (for example, https://github.com/JuanitoFatas/fast-ruby#arraybsearch-vs-arrayfind-code). Those I believe will remain with a clear advantage for one of the variants for a long time.
  • Other benchmarks, where the difference is minimal, and highly relies on the specific Ruby implementation and version, and where the slow and fast variants might switch regularly.

I think the second category deserves a clear warning that those results were measured on some version of CRuby and might not apply anymore, and likely do not apply to other Ruby implementations.

For fun, @gogainda ran these benchmarks on TruffleRuby at https://github.com/gogainda/fast-truffleruby
What I can see from a quick look is many of the differences on MRI don't exist on TruffleRuby (e.g., Sequential vs Parallel Assignment).
Also, many of these micro benchmarks optimize away (>1 billion i/s), i.e., in other words doing that operation alone costs basically nothing or like <10 cycles, which I interpret as a useful word of caution against microbenchmarks which might test something real code wouldn't, and might show differences that don't matter in practice.
I'd recommend in general to benchmark in the setup of your app/program, on the machine where the performance will matter. For example, a variant might give be 25% faster in a microbenchmark, but yield a 0% speedup on the full app and therefore be of limited value.

Dominant language
Ruby
Stars
5.7k
Forks
370
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fastruby/fast-ruby

All issues in fastruby/fast-ruby

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.