Track benchmarks over time in CI

Ouverte
  • Quelque chose ne va pas ?
#298 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
18/100
Type d'issue
Fonctionnalité
Clarté
À clarifier
Activité
Active
Stack technique
github-actions, python, rust
Domaine
ci-cd, performance

Piste de recherche

This is a design proposal, not a task yet. Start with the benchmark results table and CSV output in _bench.py, which the issue says already record step times and errors against JAX, and with the performance targets on #285. The first decision is which tool to adopt (pytest-benchmark, criterion.rs, github-action-benchmark, codspeed or bencher) and how much noise the thresholds must tolerate. Done would mean an agreed plan from a maintainer, then a first CI job that records one benchmark over time.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement ml

🌀 Claude: The ML demos' numbers are measured by hand and posted on #285, with the CPU and versions. Nothing records them over time, so a regression in xarray-sql, ddx, einfold or DataFusion shows up only when someone measures again.

Proposal: track benchmarks over time in CI, starting small:

  • pytest-benchmark for the Python benchmarks, including the ML demos at fixed sizes. Each step's time and its error against JAX are already in _bench.py's results table and CSV.
  • criterion.rs where the Rust code needs it.
  • github-action-benchmark to store the history and flag a regression on a PR.

Hosted runners vary from run to run, so the thresholds need to allow for that noise. Alternatives: codspeed.io, which counts instructions rather than measuring time, and bencher.dev. Both are free for open source today, but carry the risk of a SaaS changing its terms. asv is a reference for what a full history and dashboard could look like.

Raised in review of #269. The performance targets themselves are tracked on #285.

Langage dominant
Python
Étoiles
142
Forks
19
Merge moyen
3 j 47 min
PR mergées (30 j)
8

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de xqlsystems/xarray-sql

Toutes les issues de xqlsystems/xarray-sql

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.