ninja-build/ninja

Add a performance test script / "performance regression tests"

Open

#1733 opened on Feb 6, 2020

View on GitHub
 (4 comments) (0 reactions) (0 assignees)C++ (10,194 stars) (1,544 forks)batch import
help wanted

Description

Currently this is done manually by building Chromium before and after a commit. I think it would be good to have a script, which you can pass two commit hashes and a path to a Chromium source, which will then compare the -d stats result.

Maybe add the functionality to misc/measure.py?

See https://github.com/ninja-build/ninja/pull/1577#issuecomment-517797439

Contributor guide

Tech stack
cpppython
Domain
testingperformancebuild system
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonNinja build system conceptsBasic understanding of performance testing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Extend the existing misc/measure.py script to accept two commit hashes and a Chromium source path. The script should build with ninja for each commit and compare the ` d stats` output. Reference the discussion in PR #1577 (comment) for context on the desired comparison logic. The deliverable is a script that automates the manual performance regression check.