Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

docs: restructure benchmarking design docs to match actual scope

オープン
  • 何か問題がありますか？
#258 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
68/100
issue の種類
ドキュメント
明瞭さ
明確に書かれている
活発さ
活発
領域
documentation

調査の方向性

Start by reading the three existing files in docs/design/ and docs/design/index.md, then inspect docs/contributing/add_benchmark.md for cross-links. Rename the existing designs, add benchmarking.md as the overview, update links and stale requirement status, and preserve the technical content while making the three-layer structure and requirement mapping clear.

索引モデルが issue の本文から書いたものです。

説明

docs: restructure benchmarking design docs to match actual scope

What is the problem with the docs

The three benchmarking design documents sound like they form a hierarchy, but they do not. A reader who starts at Benchmarking System expects an umbrella design. What they get is the execution and operations design, written first, with later work parked in two other files whose names sound like "more metadata."

That mismatch shows up in three ways:

  1. Names do not match scope. "System" claims everything. "Integration" and "Metadata convention" both sound like how metadata is attached to packages, so it is hard to tell them apart.
  2. The two-tier story is incomplete. benchmark_system.md §1.1 says the architecture is two-tiered (ado experiments vs Nexus integration). A third concern — logical benchmarks, instances, and result aggregation — has no place in that picture, and the detailed "Tier 2" lives in another file that does not say it is Tier 2.
  3. Cross-links and leftovers are stale. benchmark_system.md still marks REQ 2.3 and REQ 7.1 as PENDING. Those decisions were later made in the integration doc. Nothing at the top of the design index tells you which file answers which requirement group.

The two later docs are also easy to confuse because both are about registering things. They register different things:

  • Run-side: benchmark experiments and benchmark submissions (runs of those experiments on a problem)
  • Problem-side: logical benchmarks and benchmark instances
Current file Title a reader expects What the file actually designs Specific issues
docs/design/benchmark_system.md The whole benchmarking design How experiments are executed and operated: ado + Ray, GitHub triggers, admin cluster, versioning conventions (REQ-1, REQ-4, REQ-5.1, REQ-6, REQ-7) Name is too broad. Mixes a high-level architecture sketch with ops detail. Still lists REQ 2.3 / REQ 7.1 as open. The two-tier table hides the later aggregation work.
docs/design/benchmark_integration_design.md How the system plugs into Nexus How to add benchmark experiments and benchmark submissions (runs of those experiments on a problem): nexus.yaml, benchmark_packages/, benchmark_submissions/space.yaml (REQ-2, REQ-3) "Integration" does not say experiments and submissions. Easy to confuse with the metadata convention, which is also about attaching metadata.
docs/design/benchmark_metadata_convention.md A naming convention for extra fields How to add logical benchmarks and benchmark instances, plus (for now) how an experiment binds to a logical benchmark so results can be aggregated (REQ-5.2 / 5.3) "Convention" undersells the schemas. Relation to the integration doc is only "builds on," not "problem-side vs run-side." Binding could later move to the experiments/submissions doc.

Proposed Solution

Treat benchmarking as one short overview plus three focused designs, named for the artifacts they cover and the requirement groups they close. Do not keep benchmark_system or benchmark_registration as filenames — the first will still be read as the whole design, and the second does not distinguish run-side registration from problem-side registration.

Proposed layout

docs/design/benchmarking.md                         overview: concepts, layers, REQ → doc map
  ├── benchmark_execution.md                        REQ-1, 4, 6, 7   (today’s system doc, trimmed)
  ├── benchmark_experiments_and_submissions.md      REQ-2, 3         (today’s integration doc)
  └── logical_benchmarks_and_instances.md           REQ-5.2 / 5.3    (today’s metadata convention)
New file Title Content
benchmarking.md Benchmarking Architecture Entry point only: terminology, three layers, and a table mapping REQ-1…7 to the child docs. No schemas.
benchmark_execution.md Benchmark Execution and Operations ado experiment packaging, Ray, result store, GitHub/admin triggers, sweep governance. Point at experiments/submissions for REQ 2.3 / 7.1 instead of leaving them PENDING.
benchmark_experiments_and_submissions.md Benchmark Experiments and Submissions Adding experiments (benchmark_packages, nexus.yaml) and submissions (benchmark_submissions/space.yaml), discovery CLI.
logical_benchmarks_and_instances.md Logical Benchmarks and Instances Adding logical benchmark definitions and instances. Experiment binding stays here for now. Metric mapping and aggregation.

The overview should state three layers so each child doc has a clear parent:

  1. Execution — ado experiment packages, Ray, result store
  2. Experiments and submissions — which experiments a package exposes, which submissions run them
  3. Logical benchmarks and instances — shared problem definition, concrete instances, comparable results

Implementation notes

  • git mv the three existing files, then add benchmarking.md as the new index.
  • Update docs/design/index.md to list the overview first, then the three children with one-line scopes.
  • Update cross-links in the design docs and docs/contributing/add_benchmark.md.
  • Do not rewrite the technical content except to fix stale PENDING items and the two-tier wording.
  • Leave experiment binding in logical_benchmarks_and_instances.md for now. It may later move to benchmark_experiments_and_submissions.md.
主要言語
Python
スター
3
フォーク
7
平均マージ
14時間 4分
マージ済み PR（30日）
15

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

IBM/algorithm-nexus のほかの issue

IBM/algorithm-nexus の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。