sharkdp/hyperfine

Excluding non-zero exit codes from Relative Time comparison.

Open

#591 opened on Dec 2, 2022

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Rust (28,110 stars) (479 forks)batch import
feature-requesthelp wanted

Description

I usually run a few sets of parameter lists which quickly explodes in number of instances (which is working as intended!). However, all the runs are shown in the summary at the end including those that exited with an non-zero exit code.

Looking at the code base, adding a filter in src/benchmark/relative_speed::compute to remove results with non-zero exit codes from the output should fix it.

I'm not really sure if there are more dependencies on the compute function elsewhere though or I would have just pushed a PR.

PS. Thanks for the all the great software! I use hyperfine, fd, and bat all the time!

Contributor guide

Tech stack
rust
Domain
cliperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Rusthyperfine codebase structurerelative speed module
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
The issue proposes adding a filter in `src/benchmark/relative speed.rs` at the `compute` function to exclude results with non zero exit codes. The contributor should examine the `compute` function to understand how results are aggregated and how to filter out entries where the exit code is non zero. Check for any other callers of `compute` that might be affected, as mentioned in the issue. The implementation should be straightforward and localized to that function.