taichi-dev/taichi

[benchmark] Use pytest-benchmark for performance regression test

Open

#1,454 opened on Jul 10, 2020

View on GitHub
 (1 comment) (2 reactions) (1 assignee)C++ (28,195 stars) (2,384 forks)batch import
feature requestgood first issuepython

Description

Concisely describe the proposed feature I'd like to make use of the pytest plugin pytest-benchmark instead of our own ad-hoc benchmarking framework which I've done with.

Describe the solution you'd like (if any) First check out the benchmarks/ directory, these are the benchmarks that we've already have. Make function names like benchmark_ -> test_. And make use of that plugin, as describe in their project page. When runing benchmark, simply use pytest benchmarks/. You may change the behavior ti benchmark to follow the same logic as ti test does.

Additional comments

Contributor guide

Tech stack
python
Domain
testingperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Python experienceFamiliarity with pytestBasic benchmarking concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
First, review the benchmarks/ directory to understand the current ad hoc framework. Then, install pytest benchmark and rename benchmark functions to test functions as per the plugin's documentation. Modify the ti command to run benchmarks using pytest. Note that the issue is currently assigned, so collaborate with the assignee if you wish to work on it.