Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

docs: restructure benchmarking design docs to match actual scope

Abierto
  • ¿Algo no está bien?
#258 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
68/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Activo
Área
documentation

Línea de trabajo

Start by reading the three existing files in docs/design/ and docs/design/index.md, then inspect docs/contributing/add_benchmark.md for cross-links. Rename the existing designs, add benchmarking.md as the overview, update links and stale requirement status, and preserve the technical content while making the three-layer structure and requirement mapping clear.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

docs: restructure benchmarking design docs to match actual scope

What is the problem with the docs

The three benchmarking design documents sound like they form a hierarchy, but they do not. A reader who starts at Benchmarking System expects an umbrella design. What they get is the execution and operations design, written first, with later work parked in two other files whose names sound like "more metadata."

That mismatch shows up in three ways:

  1. Names do not match scope. "System" claims everything. "Integration" and "Metadata convention" both sound like how metadata is attached to packages, so it is hard to tell them apart.
  2. The two-tier story is incomplete. benchmark_system.md §1.1 says the architecture is two-tiered (ado experiments vs Nexus integration). A third concern — logical benchmarks, instances, and result aggregation — has no place in that picture, and the detailed "Tier 2" lives in another file that does not say it is Tier 2.
  3. Cross-links and leftovers are stale. benchmark_system.md still marks REQ 2.3 and REQ 7.1 as PENDING. Those decisions were later made in the integration doc. Nothing at the top of the design index tells you which file answers which requirement group.

The two later docs are also easy to confuse because both are about registering things. They register different things:

  • Run-side: benchmark experiments and benchmark submissions (runs of those experiments on a problem)
  • Problem-side: logical benchmarks and benchmark instances
Current file Title a reader expects What the file actually designs Specific issues
docs/design/benchmark_system.md The whole benchmarking design How experiments are executed and operated: ado + Ray, GitHub triggers, admin cluster, versioning conventions (REQ-1, REQ-4, REQ-5.1, REQ-6, REQ-7) Name is too broad. Mixes a high-level architecture sketch with ops detail. Still lists REQ 2.3 / REQ 7.1 as open. The two-tier table hides the later aggregation work.
docs/design/benchmark_integration_design.md How the system plugs into Nexus How to add benchmark experiments and benchmark submissions (runs of those experiments on a problem): nexus.yaml, benchmark_packages/, benchmark_submissions/space.yaml (REQ-2, REQ-3) "Integration" does not say experiments and submissions. Easy to confuse with the metadata convention, which is also about attaching metadata.
docs/design/benchmark_metadata_convention.md A naming convention for extra fields How to add logical benchmarks and benchmark instances, plus (for now) how an experiment binds to a logical benchmark so results can be aggregated (REQ-5.2 / 5.3) "Convention" undersells the schemas. Relation to the integration doc is only "builds on," not "problem-side vs run-side." Binding could later move to the experiments/submissions doc.

Proposed Solution

Treat benchmarking as one short overview plus three focused designs, named for the artifacts they cover and the requirement groups they close. Do not keep benchmark_system or benchmark_registration as filenames — the first will still be read as the whole design, and the second does not distinguish run-side registration from problem-side registration.

Proposed layout

docs/design/benchmarking.md                         overview: concepts, layers, REQ → doc map
  ├── benchmark_execution.md                        REQ-1, 4, 6, 7   (today’s system doc, trimmed)
  ├── benchmark_experiments_and_submissions.md      REQ-2, 3         (today’s integration doc)
  └── logical_benchmarks_and_instances.md           REQ-5.2 / 5.3    (today’s metadata convention)
New file Title Content
benchmarking.md Benchmarking Architecture Entry point only: terminology, three layers, and a table mapping REQ-1…7 to the child docs. No schemas.
benchmark_execution.md Benchmark Execution and Operations ado experiment packaging, Ray, result store, GitHub/admin triggers, sweep governance. Point at experiments/submissions for REQ 2.3 / 7.1 instead of leaving them PENDING.
benchmark_experiments_and_submissions.md Benchmark Experiments and Submissions Adding experiments (benchmark_packages, nexus.yaml) and submissions (benchmark_submissions/space.yaml), discovery CLI.
logical_benchmarks_and_instances.md Logical Benchmarks and Instances Adding logical benchmark definitions and instances. Experiment binding stays here for now. Metric mapping and aggregation.

The overview should state three layers so each child doc has a clear parent:

  1. Execution — ado experiment packages, Ray, result store
  2. Experiments and submissions — which experiments a package exposes, which submissions run them
  3. Logical benchmarks and instances — shared problem definition, concrete instances, comparable results

Implementation notes

  • git mv the three existing files, then add benchmarking.md as the new index.
  • Update docs/design/index.md to list the overview first, then the three children with one-line scopes.
  • Update cross-links in the design docs and docs/contributing/add_benchmark.md.
  • Do not rewrite the technical content except to fix stale PENDING items and the two-tier wording.
  • Leave experiment binding in logical_benchmarks_and_instances.md for now. It may later move to benchmark_experiments_and_submissions.md.
Lenguaje dominante
Python
Estrellas
3
Forks
7
Merge medio
14 h 4 min
PR fusionados (30 d)
15

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de IBM/algorithm-nexus

Todos los issues de IBM/algorithm-nexus

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.