Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

`darnit-reproducibility`: reproducibility layer + tightness taxonomy

Aperta
  • Qualcosa non va?
#415 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
tooling

Direzione di ricerca

Start by comparing the taxonomy needs in this issue with #393 (ontology) and #227 (HermeticBuild), then review the declaration options: .project/ context and [reproducibility] in .baseline.toml. Clarify the declaration format and how existing controls map to layer and tightness; done means the taxonomy handles compound layers and the third-party-binary case and can serialize into the ontology vocabulary.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

module-reproducibility

Summary

"Reproducible" is not a single concept. Auditing meaningfully requires two
independent axes:

  1. Layer being reproduced.
  2. Tightness of match at that layer.

Without this taxonomy, users cannot declare what they are aiming for and
darnit cannot audit meaningfully -- the required signals differ per (layer,
tightness) pair. A repo can be statistically reproducible without pinning
PYTHONHASHSEED; a regulatory-grade forensic tool cannot. Conflating these
under one "reproducibility" label produces false PASS or false FAIL,
depending on which direction the assumption drifts.

Layers to name explicitly

  • Layer 1: Source -> built tool. Deterministic compilation,
    SOURCE_DATE_EPOCH, pinned toolchain, container-build digest. This is
    build reproducibility; overlaps with RE-02.01 HermeticBuild (#227).
  • Layer 2: Built tool + inputs -> outputs. Pinned execution
    environment, seeded RNG, deterministic ops, workflow spec, data
    lineage. This is what most people mean by "reproducible experiment."
  • Layer 3: Outputs -> conclusions. Downstream statistical processing,
    meta-analysis, plot generation. The layer where analytical
    reproducibility (same inputs -> same conclusions under the declared
    test) lives.

Compound case the taxonomy must handle: source that compiles to a tool
(Layer 1) that then runs the experiment (Layer 2). Two layers, potentially
different tightness tiers each. Similarly, experiments that consume a
third-party binary the researcher did not build -- Layer 1 is somebody
else's problem, but Layer 2 remains auditable.

Tightness tiers to name explicitly

  • Bit-for-bit: byte-equal outputs. Regulatory, forensic,
    cryptographic, hermetic-build contexts.
  • Within tolerance: outputs statistically equivalent under a declared
    epsilon or distributional test (KS, chi-square, etc.). The honest bar
    for stochastic ML, MCMC, ensemble methods, sampling, GANs.
  • Runs-to-completion: environment + data available; produces some
    result of the declared type. The honest bar for most published
    notebooks and the floor darnit should never audit below.

Scope

  • Formalize the (layer, tightness) two-axis taxonomy.
  • Design how a repo declares its target: candidate surfaces are a new key
    in .project/ context or a [reproducibility] block in
    .baseline.toml. Decide during clarify.
  • Map darnit-reproducibility's existing controls onto (layer, tightness)
    so each control knows which target it audits against.

Non-goals for v0

  • The specific signal checks (see #416).
  • Auto-remediation of missing tier declarations.
  • Rewriting existing controls to be tier-aware in the same PR -- tag
    with a default tier and re-tag in a follow-up.

Constitution alignment

  • II. Conservative-by-Default: if a repo declares "bit-for-bit" but
    a control at that tier is missing evidence, WARN -- never silent PASS.
  • IV. Never Guess User Values: the (layer, tightness) declaration is
    a user-judgment key. darnit MAY propose a tier from repo heuristics
    (e.g., Dockerfile with digest -> likely bit-for-bit at Layer 1) but
    MUST NOT conclude it.
  • III. TOML-First: tier declaration lives in TOML or .project/,
    not in Python.

Related

  • #414 (extractor module).
  • #416 (heuristic checks -- each control declares its (layer, tightness)
    applicability).
  • #393 (ontology) -- the taxonomy must serialize into the ontology
    vocabulary.
  • #227 (HermeticBuild) -- the canonical Layer 1 / bit-for-bit control.
Lingua principale
Python
Stelle
22
Fork
27
Merge medio
1g 20h
PR unite (30g)
39

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di darnitdevorg/darnit

Tutte le issue di darnitdevorg/darnit

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.