Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Decide how to collect and publish validation strategies and benchmarks

Open
  • Is something wrong?
#29 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
jupyter-notebook, python
Domain
documentation, performance, testing-qa

Research direction

Start by inventorying devguide/validation/ and reading the existing profiling notebook, measurement definitions, raw reports, and devguide/DECISIONS.md. Compare the listed publication routes against the acceptance criteria, then record the chosen route, evidence lifecycle, provenance, maintenance responsibilities, and revisit trigger in devguide/DECISIONS.md; done means the decision is documented and existing cases are linked.

Written by the indexing model from the issue text.

Description

proposal

What

Decide how DockingMT should collect, document, maintain and publish its validation strategies and benchmark evidence as the project grows. This is a future design decision; opening this issue does not select a publishing platform or authorize building a website.

Current evidence

Validation and performance material already lives in devguide/validation/: exploratory 181L redocking, external 1IEP PDBQT checks, preparation audits, and workflow phase profiling. The recent work in #28 added an executed profiling notebook, measurement definitions and limits, and versioned raw reports. Preservation is in place, but the long-term reader-facing organization and publication route remain undecided.

Options to compare

  • A “Validation and benchmarks” section in the main documentation, with rendered notebooks and links to retained data.
  • A dedicated website for validation/benchmark evidence, linked from the main documentation.
  • A staged or hybrid route: maintain source material in the repository, start with a documentation section, and split the presentation later if scale or reader needs justify it.

Evaluate discoverability, audience, reproducibility, versioned historical results, hosting/build complexity, maintenance cost, and avoiding multiple copies of the same evidence. Keep source records independent of the chosen presentation so changing publication routes preserves history.

Questions to resolve

  • How should validation strategies, reference cases, test implementation, raw outputs, notebooks, benchmark summaries and published claims relate to each other?
  • How should scientific correctness/accuracy validation be distinguished from software regression, integration qualification and runtime/memory benchmarks? Passing CI or exploratory preparation must not imply scientific validation.
  • What evidence should a publishable case retain: input/dataset origin and hashes, method/backend and versions, preparation/state choices, units and metric definitions, tolerances, random seeds, hardware/thread resources, repeated samples, limitations and failure/negative cases?
  • Which material belongs in Git, and when would large artifacts need an external versioned store? How should notebooks, scripts and generated pages reuse the same data?
  • How should readers distinguish historical baselines, current results, provisional assessments and scientifically qualified results? What makes two runs comparable?
  • Who maintains the collection, when are cases rerun, and what lightweight checks or separately scheduled expensive runs would keep links, notebooks and claims current?
  • What is the smallest initial publication route, and what concrete need would justify a separate site?

Scope

DockingMT owns this decision and its scientific evidence. Preserve existing cases and molecular ownership. No MolSysMT/MolSysViewer changes or coordination are needed to open or develop this local proposal. If the eventual design proposes a shared suite publishing rule or reusable cross-repository tooling, take that separate proposal to MolSysSuite.

Acceptance criteria

  • Inventory the current material and identify its intended readers.
  • Compare the publication options and record the chosen route, rationale and any trigger for revisiting it.
  • Define a minimal evidence organization and lifecycle, including reproducibility, interpretation, historical provenance and maintenance responsibilities.
  • Keep a single maintained source of evidence and distinguish scientific validation from performance measurements and software checks.
  • Record the decision in devguide/DECISIONS.md, link existing cases and open bounded implementation follow-ups if needed. A complete website or automatic benchmark service is not required to close this decision issue.

Related: #28 (architecture/performance evidence), #22 (coverage evidence, a distinct software metric).

Dominant language
Python
Stars
1
Forks
0
Avg merge
30m
Merged PRs (30d)
1

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from uibcdf/dockingmt

All issues in uibcdf/dockingmt

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.