Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Benchmark every PR on deterministic costs, with on-demand stable-hardware runs and a results history

Aperta
  • Qualcosa non va?
#90 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
docker, github-actions, go
Ambito
ci-cd, performance, testing-qa

Direzione di ricerca

Start by inventorying the Go micro-benchmarks, tests/performance, tests/stress, tests/resources, tests/enry-performance, and BenchmarkProfileScanner, then inspect how CI currently runs them. Use the related #89 and #75 proposals to clarify dependencies. Done means the three benchmark tiers, matching local target, public history and regression reporting are implemented without a schedule trigger.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement follow-up

Why

The raw material exists, but nothing watches performance continuously:

  • 16 Go micro-benchmarks, none run in CI;
  • BenchmarkProfileScanner, with pprof capture, disabled unless DIRCUE_PROFILE_ROOT is set;
  • Docker-based harnesses (tests/performance against Linguist, tests/stress, tests/resources with cgroup limits, tests/enry-performance), all run by hand, with results committed as JSON.

There is no history, no baseline comparison and no alerting. That's how the Git-path cliff (#73) shipped unnoticed: 23.5 s vs 6.5 s for the language pass, and 205.6 s vs 6.7 s for metrics, on the Linux kernel.

Shared CI runners are too noisy for wall-time gates (±20% is common), so the design splits into what is deterministic and what needs stable hardware. All of it is event-driven: nothing runs on a schedule (#92).

Proposal: three tiers

  1. Every PR (hosted runners, noise-free signals).
    • Logical-work counters (#89) on small pinned fixtures, gated against committed budgets. Physical costs are gated only under a pinned pack layout and controlled cache conditions, e.g. a small delta-replay fixture with an explicit byte-amplification budget.
    • Go micro-benchmarks compared with benchstat against the merge base, gating on allocs/op and B/op, which are stable on shared runners.
    • Wall time reported but not gated.
  2. On demand (stable hardware). A workflow_dispatch-only workflow, plus a local make target that produces the same result format. It targets a self-hosted runner label when a dedicated, quiet machine exists (self-hosted runners are free for public repositories), and runs macro-benchmarks on the golden corpus (#75):
    • N repeated runs per case, covering both source modes (Git and directory), each preset, and key modules;
    • wall time, CPU, peak RSS and bytes read;
    • pprof profiles archived.
    • Each run appends to a public history (e.g. github-action-benchmark to GitHub Pages, or Bencher). Regressions are detected statistically against the previous recorded runs and reported on the run, or by opening an issue.
  3. Release.
    • The full matrix against the previous release, Linguist and scc.
    • Constrained-resource runs (cgroup memory and CPU limits) that prove each preset's and memory ceiling's claims.
    • BENCHMARKS.md regenerated from data.

Budgets. Publish per-corpus-entry targets for reference hardware, e.g. the kernel language pass from Git, peak RSS, and analyze all time, and check them at release.

Acceptance

  • PR-tier counter and allocation gates live in CI.
  • On-demand stable-hardware workflow (workflow_dispatch only) and a matching local target, feeding a public history with regression detection. No schedule: trigger.
  • Release-tier report generated automatically and attached to each release.
  • Existing harnesses (tests/performance, tests/resources, BenchmarkProfileScanner) folded into the tiers rather than duplicated.

Related: #39 (optimization campaign methodology), #73.

Part of the 1.0 quality program: #83. Epic: #81.

Lingua principale
Go
Stelle
0
Fork
0
Merge medio
4h 47m
PR unite (30g)
52

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di war-and-code/dircue

Tutte le issue di war-and-code/dircue

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.