How to specify setup code with Runner.bench_func() ?

Open
  • Is something wrong?
#38 7 comments 4 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
performance, tooling

Research direction

Start with Runner.bench_func() and the subprocess(main) behavior shown in the supplied example; also read the related issue 28. Reproduce the repeated BenchModule construction, then verify that the requested setup runs once across benchmark runs while the benchmarks still execute normally.

Written by the indexing model from the issue text.

Description

I am trying to bench some data structures with a large number of items in each structure. I would like to have a single setup that is called once (and only once) for all benchmark runs, because the setup takes several seconds to complete.

While debugging, I discovered this very peculiar behavior that perf reloads the entire main module for each run.

import perf
import time

def run_benchmarks(module):
    runner = perf.Runner()

    def get_doc(attr):
        return getattr(module, attr).__doc__

    bench_names = sorted([ x for x in dir(module) if x.startswith('bench_') ], key=get_doc)
    for bench_name in bench_names:
        bench = getattr(module, bench_name)
        doc = bench.__doc__
        print('Running: %s' % (doc))
        runner.bench_func(doc, bench)

class BenchModule(object):
    def __init__(self):
        print('Creating BenchModule()')
        time.sleep(3)
        print('Created BenchModule()')

    def bench_foo(self):
        '''Benchmark: foo'''
        return list(range(1000))

    def bench_bar(self):
        '''Benchmark: bar'''
        return list(range(1000))

if __name__ == '__main__':
    print('Starting benchmarks')
    module = BenchModule()
    run_benchmarks(module)

When I run this, I see output like the following:

Starting benchmarks
Creating BenchModule()
Created BenchModule()
Running: Benchmark: bar
Starting benchmarks
Creating BenchModule()
Created BenchModule()
Running: Benchmark: bar
Running: Benchmark: foo
.Starting benchmarks
Creating BenchModule()
Created BenchModule()
Running: Benchmark: bar
Running: Benchmark: foo
.Starting benchmarks
Creating BenchModule()
Created BenchModule()
Running: Benchmark: bar
Running: Benchmark: foo
.Starting benchmarks
Creating BenchModule()

[ ... snip ... ]

.Starting benchmarks
Creating BenchModule()
Created BenchModule()
Running: Benchmark: bar
Running: Benchmark: foo
.
Benchmark: foo: Mean +- std dev: 9.83 us +- 0.16 us

... what the ? 😖

Why is it printing my Starting benchmarks line more than once? More importantly, how is it doing this? Is there some black magic going on with subprocess(__main__)?

But back to the original issue that lead me down this rathole; how do I prevent perf from running my BenchModule constructor on each run? I put a 3-second sleep in there to illustrate why the current behavior is obnoxious. In my real world benchmark, the setup time is more than 30 seconds, and running the complete test suite lasts a few hours.

This is possibly related: https://github.com/vstinner/perf/issues/28

Dominant language
Python
Stars
967
Forks
112
Avg merge
5d 8h
Merged PRs (30d)
9

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from psf/pyperf

All issues in psf/pyperf

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.