perf(bench): add a production readEvents scaling benchmark

オープン
#2,476 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
30/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
nodejs, typescript

調査の方向性

Start from the existing runner in scripts/run-benchmarks.mjs and the built-in cases in packages/squad-sdk/src/runtime/benchmarks.ts, then find LocalSquadBackend.readEvents for the local file-state backend. Add the six fixture cases (100/1,000/10,000 events, plus limit 1 with one-event and empty tails) to the existing runner and keep its output shape compatible. Done when the built method runs with 2 warmups and 3 rounds of 10 timed samples, a deliberately broken result fails the run, and a test-only 2x slowdown exceeds the comparison threshold.

索引モデルが issue の本文から書いたものです。

説明

Summary

Add one production readEvents scaling workload to the existing performance runner. The current built-in configLoad, charterCompile, routing, model-selection and export/import cases largely simulate JSON/string/regex work rather than invoke the corresponding product implementations.

Proposed primary owner: unit-contract-tester.

Evidence and scope

Audited revision: 47082a81799710c3e457b594123d190c00596ebf, observed 2026-11-12.

  • Built-in benchmark cases were exercised for three baseline runs of 1,000 iterations per case. Their output is not evidence of equivalent production-path performance.
  • Existing entry points include scripts/run-benchmarks.mjs and scripts/measure-cold-start.mjs. No workflow invocation of those performance runners was found at the audited revision. Smoke/output-shape tests do not demonstrate sensitivity to realistic product regressions.
  • The local packaged build could not run: TypeScript/Vitest were unavailable and locked dependency restoration returned a configured-feed E404 for hono-4.13.17.tgz. Missing dist also blocked the built benchmark and cold-start runners. This is a measurement limitation, not evidence that the CLI is slow.

The audit's fallback used Node's built-in TypeScript transformation and isolated relative-import resolution. Those function-level diagnostics exclude import/setup time and must not certify packaged startup, application throughput, total allocations, token efficiency, or whole-agent-session memory.

Scope

Parent: github/squad#2481.

One production API only: LocalSquadBackend.readEvents with the local file-state backend. Add a fixture/case to the existing runner foundation; do not build a new benchmark framework. Packaged help/version startup is split into github/squad#2482.

Out of scope: config, charter, routing, model selection, memory search, shell persistence, backend optimization, CI gates, nightly automation and cross-platform rollout. The optimization belongs to github/squad#2479.

Measurable acceptance and tests

  • Run the built production method, not a JSON/string simulation or source-loader shim. Fixture sizes: 100/1,000/10,000 valid events; limit: 1 with one-event-tail and empty-tail cursors, six cases total.
  • For each case, collect three rounds of ten timed observations after two untimed warmups. Exclude fixture/import setup and retain all 30 raw times plus p50/p95, exact SHA, Node/platform/build mode and cache policy.
  • Assert event count, final sequence, cursor and resumability in the fixtures. A deliberately broken result must fail the run rather than produce a successful timing.
  • Deterministic output tests cover sample counts/quantiles, nonfinite or missing data, fixture failure and a nonzero runner exit. Existing runner consumers/output contracts remain compatible.
  • Exercise the comparison/reporting path with a calibrated test-only slowdown of at least 2x; its reported median ratio must exceed the declared comparison threshold. This proves regression sensitivity, not production optimization benefit; do not add a blocking CI threshold in this issue.
  • Check in the portable fixture and document one exact build/run command with a passing sample artifact. Missing dependencies/dist remain a failing or blocked measurement, never a successful substitute.

Done means one trustworthy six-case production benchmark and its runner/result tests, not coverage of every product path.

Related work and non-goals

Follow-up to the already-merged runner foundation github/squad#1111, not another request to add it. github/squad#2449 is a distinct model-quality paired A/B workload.

Methodology: dotnet/aspnetcore#70219, microsoft/waza#846, and the packaged-path correction in microsoft/aspire#16704. No product optimization or end-to-end speedup has been demonstrated by this issue.

主要言語
TypeScript
スター
3.4k
フォーク
516
平均マージ
1日 15時間
マージ済み PR(30日)
116

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

github/squad のほかの issue

github/squad の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。