Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

docs: restructure benchmarking design docs to match actual scope

未关闭
  • 哪里不对？
#258 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
68/100
Issue 类型
文档
描述清晰度
描述清楚
活跃度
活跃
领域
documentation

调研方向

Start by reading the three existing files in docs/design/ and docs/design/index.md, then inspect docs/contributing/add_benchmark.md for cross-links. Rename the existing designs, add benchmarking.md as the overview, update links and stale requirement status, and preserve the technical content while making the three-layer structure and requirement mapping clear.

由索引模型根据 Issue 内容生成。

描述

docs: restructure benchmarking design docs to match actual scope

What is the problem with the docs

The three benchmarking design documents sound like they form a hierarchy, but they do not. A reader who starts at Benchmarking System expects an umbrella design. What they get is the execution and operations design, written first, with later work parked in two other files whose names sound like "more metadata."

That mismatch shows up in three ways:

  1. Names do not match scope. "System" claims everything. "Integration" and "Metadata convention" both sound like how metadata is attached to packages, so it is hard to tell them apart.
  2. The two-tier story is incomplete. benchmark_system.md §1.1 says the architecture is two-tiered (ado experiments vs Nexus integration). A third concern — logical benchmarks, instances, and result aggregation — has no place in that picture, and the detailed "Tier 2" lives in another file that does not say it is Tier 2.
  3. Cross-links and leftovers are stale. benchmark_system.md still marks REQ 2.3 and REQ 7.1 as PENDING. Those decisions were later made in the integration doc. Nothing at the top of the design index tells you which file answers which requirement group.

The two later docs are also easy to confuse because both are about registering things. They register different things:

  • Run-side: benchmark experiments and benchmark submissions (runs of those experiments on a problem)
  • Problem-side: logical benchmarks and benchmark instances
Current file Title a reader expects What the file actually designs Specific issues
docs/design/benchmark_system.md The whole benchmarking design How experiments are executed and operated: ado + Ray, GitHub triggers, admin cluster, versioning conventions (REQ-1, REQ-4, REQ-5.1, REQ-6, REQ-7) Name is too broad. Mixes a high-level architecture sketch with ops detail. Still lists REQ 2.3 / REQ 7.1 as open. The two-tier table hides the later aggregation work.
docs/design/benchmark_integration_design.md How the system plugs into Nexus How to add benchmark experiments and benchmark submissions (runs of those experiments on a problem): nexus.yaml, benchmark_packages/, benchmark_submissions/space.yaml (REQ-2, REQ-3) "Integration" does not say experiments and submissions. Easy to confuse with the metadata convention, which is also about attaching metadata.
docs/design/benchmark_metadata_convention.md A naming convention for extra fields How to add logical benchmarks and benchmark instances, plus (for now) how an experiment binds to a logical benchmark so results can be aggregated (REQ-5.2 / 5.3) "Convention" undersells the schemas. Relation to the integration doc is only "builds on," not "problem-side vs run-side." Binding could later move to the experiments/submissions doc.

Proposed Solution

Treat benchmarking as one short overview plus three focused designs, named for the artifacts they cover and the requirement groups they close. Do not keep benchmark_system or benchmark_registration as filenames — the first will still be read as the whole design, and the second does not distinguish run-side registration from problem-side registration.

Proposed layout

docs/design/benchmarking.md                         overview: concepts, layers, REQ → doc map
  ├── benchmark_execution.md                        REQ-1, 4, 6, 7   (today’s system doc, trimmed)
  ├── benchmark_experiments_and_submissions.md      REQ-2, 3         (today’s integration doc)
  └── logical_benchmarks_and_instances.md           REQ-5.2 / 5.3    (today’s metadata convention)
New file Title Content
benchmarking.md Benchmarking Architecture Entry point only: terminology, three layers, and a table mapping REQ-1…7 to the child docs. No schemas.
benchmark_execution.md Benchmark Execution and Operations ado experiment packaging, Ray, result store, GitHub/admin triggers, sweep governance. Point at experiments/submissions for REQ 2.3 / 7.1 instead of leaving them PENDING.
benchmark_experiments_and_submissions.md Benchmark Experiments and Submissions Adding experiments (benchmark_packages, nexus.yaml) and submissions (benchmark_submissions/space.yaml), discovery CLI.
logical_benchmarks_and_instances.md Logical Benchmarks and Instances Adding logical benchmark definitions and instances. Experiment binding stays here for now. Metric mapping and aggregation.

The overview should state three layers so each child doc has a clear parent:

  1. Execution — ado experiment packages, Ray, result store
  2. Experiments and submissions — which experiments a package exposes, which submissions run them
  3. Logical benchmarks and instances — shared problem definition, concrete instances, comparable results

Implementation notes

  • git mv the three existing files, then add benchmarking.md as the new index.
  • Update docs/design/index.md to list the overview first, then the three children with one-line scopes.
  • Update cross-links in the design docs and docs/contributing/add_benchmark.md.
  • Do not rewrite the technical content except to fix stale PENDING items and the two-tier wording.
  • Leave experiment binding in logical_benchmarks_and_instances.md for now. It may later move to benchmark_experiments_and_submissions.md.
主要语言
Python
星标
3
派生
7
平均合并
14 小时 4 分钟
30 天内合并 PR
15

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

IBM/algorithm-nexus 的其他 Issue

查看 IBM/algorithm-nexus 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。