The benchmark does not record how long each fit takes

Open Beginner friendly
#1,329 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Read faircode/benchmark.py at run_audit and collect_provenance, then inspect the progress_callback and provenance-writing paths in faircode/cli.py. Use a stub clock in the relevant test to check per-combination timing values. Done means each run records those seconds in PROVENANCE.json, diff_results.py ignores the added timings, and the existing schema check passes.

Written by the indexing model from the issue text.

Description

enhancement reproducibility tooling

Where: faircode/benchmark.py (run_audit, collect_provenance), faircode/cli.py.

Current gap: The only timing in a benchmark run is what the user sees on the clock: a full run takes 5 to 15 minutes and the in-processing fit on Healthcare Readmission alone about 98 s (closed as #1238), but nothing records which (audit, strategy, model) combination took how long, so a slowdown after a dependency bump (see the rerun-benchmark label) cannot be located.

Suggested approach: Time each combination with time.perf_counter in run_audit and write the seconds into the existing progress_callback call and a timings list in the new run's PROVENANCE.json entry (additive, so the existing schema check keeps passing).

Done when: a run writes per-combination seconds into PROVENANCE.json, diff_results.py ignores them, and a test with a stub clock checks the values.

Dominant language
HTML
Stars
47
Forks
45
Avg merge
2h 49m
Merged PRs (30d)
69

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from yakew7/Fair-Code

All issues in yakew7/Fair-Code

Similar issues

More Testing & QA issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.