feat(bench): track benchmark and memory results over time instead of a single-run gate
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 12/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 冷清
- 技术栈
- github, github-actions, go
调研方向
Start with the regression job in .github/workflows/bench.yml and the baseline in tools/bench/baseline.json, then find CheckRegression to see why it stops at the first failure. The issue asks for an OpenSpec change to PERF-02 and INDX-06 before any code, and the tool choice (github-action-benchmark or Bencher) is left to the spec. Done means a spec is agreed, the tool is picked, and a per-phase metrics output is designed, which is design work rather than a first contribution.
由索引模型根据 Issue 内容生成。
描述
What exists today?
The regression job in .github/workflows/bench.yml runs each night. It indexes the synthetic corpus three times, takes the median, and compares it with one stored baseline in tools/bench/baseline.json. It fails at a 10% throughput drop, at 15% more peak RSS, or above 4 GiB. It never blocks a merge. The publish job reports absolute numbers each week.
How does it behave now?
The gate measures the host as much as the code, and it keeps no history.
- Two runs on the same commit differed by 21% in throughput (#252).
- After #250, the three trials of one night spread by up to 12% in peak RSS.
- The job failed every night from 3 to 7 November. On three of those nights, its message named only throughput, because
CheckRegressionreturns at its first failure. The 40% RSS growth from #250 stayed hidden. - No one can see when a number started to move, or which phase of an index moved it. The #265 investigation rebuilt that by hand from profiles.
How should it behave?
- Each nightly run and each dispatch records its results as a time series. The series holds throughput, peak RSS, query latency, node and edge counts, and numbers per phase of an index (extract, resolve, write): bytes allocated, live heap at the end of the phase, GC cycles and batch bytes.
- The binary reports the per-phase numbers from
runtime/metrics, so the runner reads them and no profile is needed. - A page shows the series as charts. An alert fires on a change that lasts over several runs, with a statistical threshold, and not on one run.
- Optional: a pull request that changes Go code gets a report of a paired comparison. The job builds the base and the head on one runner, alternates their trials, and compares them with
benchstat. The report does not block the merge. - The tool comes from OSS. The candidates are
github-action-benchmarkand Bencher. The spec picks one, and the implementation checks its current version.
This issue replaces #252.
What kind of improvement?
Performance
Would this change existing behavior?
Yes. The nightly job stops being a pass or fail result from one run against one baseline. That changes PERF-02 and INDX-06, so it needs an OpenSpec change first. If the binary reports per-phase numbers through a new flag or output, that flag is a change to the CLI surface, and the spec names it.
🤖 Generated with Claude Code
- 主要语言
- Go
- 星标
- 0
- 派生
- 0
- 平均合并
- 7 小时 42 分钟
- 30 天内合并 PR
- 111
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
seanb4t/codegraph-go 的其他 Issue
-
chore needs-triage
难度 1/5 1 小时以内 新手友好度 88/100
seanb4t/codegraph-go#380 ·
维护者通常 1 天内回复
-
chore needs-triage
难度 2/5 1-3 小时 新手友好度 82/100
seanb4t/codegraph-go#379 ·
维护者通常 1 天内回复
-
area:query chore needs-triage spec-needed
难度 2/5 1-3 小时 新手友好度 74/100
seanb4t/codegraph-go#377 ·
维护者通常 1 天内回复
-
chore needs-triage
难度 1/5 1-3 小时 新手友好度 84/100
seanb4t/codegraph-go#362 ·
维护者通常 1 天内回复
-
area:query bug needs-triage
难度 2/5 1-3 小时 新手友好度 80/100
seanb4t/codegraph-go#355 ·
维护者通常 1 天内回复
查看 seanb4t/codegraph-go 的全部 Issue
相似的 Issue
-
go-vulnerability security-scan
难度 1/5 1 小时以内 新手友好度 84/100
teslashibe/agent-go#35 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
alecthomas/chroma#1498 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 83/100
维护者通常 8 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
-
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复