ggml-org/whisper.cpp

Should we use process time for benchmarks/timings?

Open

#382 opened on Jan 7, 2023

View on GitHub
 (2 comments) (0 reactions) (0 assignees)C++ (5,535 forks)batch import
enhancementgood first issue

Repository metrics

Stars
 (49,693 stars)
PR merge metrics
 (Avg merge 5d 20h) (36 merged PRs in 30d)

Description

Currently, the bench tool simply spits out the timings from whisper_print_timings. These timings are not process specific and as such are influenced by other processes on the system.

Should we use something like CLOCK_PROCESS_CPUTIME_ID instead of CLOCK_MONOTONIC in ggml_time_*? This would make the benchmarks in #89 outdated, but would make future benchmarks more realistic.

Contributor guide

Research direction
Investigate the current use of CLOCK MONOTONIC in ggml time * functions and compare with CLOCK PROCESS CPUTIME ID to assess impact on benchmark accuracy. Consider platform compatibility and thread safety.
Tech stack
cpp
Domain
performance
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
None
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75