feat(bench): track benchmark and memory results over time instead of a single-run gate
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- github, github-actions, go
- Domain
- build-system, observability, performance
Research direction
Start with the regression job in .github/workflows/bench.yml and the baseline in tools/bench/baseline.json, then find CheckRegression to see why it stops at the first failure. The issue asks for an OpenSpec change to PERF-02 and INDX-06 before any code, and the tool choice (github-action-benchmark or Bencher) is left to the spec. Done means a spec is agreed, the tool is picked, and a per-phase metrics output is designed, which is design work rather than a first contribution.
Written by the indexing model from the issue text.
Description
What exists today?
The regression job in .github/workflows/bench.yml runs each night. It indexes the synthetic corpus three times, takes the median, and compares it with one stored baseline in tools/bench/baseline.json. It fails at a 10% throughput drop, at 15% more peak RSS, or above 4 GiB. It never blocks a merge. The publish job reports absolute numbers each week.
How does it behave now?
The gate measures the host as much as the code, and it keeps no history.
- Two runs on the same commit differed by 21% in throughput (#252).
- After #250, the three trials of one night spread by up to 12% in peak RSS.
- The job failed every night from 3 to 7 November. On three of those nights, its message named only throughput, because
CheckRegressionreturns at its first failure. The 40% RSS growth from #250 stayed hidden. - No one can see when a number started to move, or which phase of an index moved it. The #265 investigation rebuilt that by hand from profiles.
How should it behave?
- Each nightly run and each dispatch records its results as a time series. The series holds throughput, peak RSS, query latency, node and edge counts, and numbers per phase of an index (extract, resolve, write): bytes allocated, live heap at the end of the phase, GC cycles and batch bytes.
- The binary reports the per-phase numbers from
runtime/metrics, so the runner reads them and no profile is needed. - A page shows the series as charts. An alert fires on a change that lasts over several runs, with a statistical threshold, and not on one run.
- Optional: a pull request that changes Go code gets a report of a paired comparison. The job builds the base and the head on one runner, alternates their trials, and compares them with
benchstat. The report does not block the merge. - The tool comes from OSS. The candidates are
github-action-benchmarkand Bencher. The spec picks one, and the implementation checks its current version.
This issue replaces #252.
What kind of improvement?
Performance
Would this change existing behavior?
Yes. The nightly job stops being a pass or fail result from one run against one baseline. That changes PERF-02 and INDX-06, so it needs an OpenSpec change first. If the binary reports per-phase numbers through a new flag or output, that flag is a change to the CLI surface, and the spec names it.
🤖 Generated with Claude Code
- Dominant language
- Go
- Stars
- 0
- Forks
- 0
- Avg merge
- 5h 16m
- Merged PRs (30d)
- 107
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from seanb4t/codegraph-go
-
area:indexer bug needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
seanb4t/codegraph-go#385 ·
Maintainers usually reply within 1 day
-
chore needs-triage
Difficulty 1/5 Under an hour Newbie friendliness 88/100
seanb4t/codegraph-go#380 ·
Maintainers usually reply within 1 day
-
chore needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
seanb4t/codegraph-go#379 ·
Maintainers usually reply within 1 day
-
chore needs-triage
Difficulty 1/5 1-3 hours Newbie friendliness 84/100
seanb4t/codegraph-go#362 ·
Maintainers usually reply within 1 day
-
area:query bug priority:low spec-needed
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
seanb4t/codegraph-go#355 ·
Maintainers usually reply within 1 day
All issues in seanb4t/codegraph-go
Similar issues
-
enhancement exporter/awskinesis needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
open-telemetry/opentelemetry-collector-contrib#53334 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
erigontech/erigon#26523 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 83/100
SocialGouv/iterion#3376 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
seaweedfs/seaweedfs#12740 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day