`darnit-reproducibility`: reproducibility layer + tightness taxonomy
メンテナーはふだん 2 日以内に返信
まだ誰も着手していません。
評価
調査の方向性
Start by comparing the taxonomy needs in this issue with #393 (ontology) and #227 (HermeticBuild), then review the declaration options: .project/ context and [reproducibility] in .baseline.toml. Clarify the declaration format and how existing controls map to layer and tightness; done means the taxonomy handles compound layers and the third-party-binary case and can serialize into the ontology vocabulary.
索引モデルが issue の本文から書いたものです。
説明
Summary
"Reproducible" is not a single concept. Auditing meaningfully requires two
independent axes:
- Layer being reproduced.
- Tightness of match at that layer.
Without this taxonomy, users cannot declare what they are aiming for and
darnit cannot audit meaningfully -- the required signals differ per (layer,
tightness) pair. A repo can be statistically reproducible without pinning
PYTHONHASHSEED; a regulatory-grade forensic tool cannot. Conflating these
under one "reproducibility" label produces false PASS or false FAIL,
depending on which direction the assumption drifts.
Layers to name explicitly
- Layer 1: Source -> built tool. Deterministic compilation,
SOURCE_DATE_EPOCH, pinned toolchain, container-build digest. This is
build reproducibility; overlaps withRE-02.01 HermeticBuild(#227). - Layer 2: Built tool + inputs -> outputs. Pinned execution
environment, seeded RNG, deterministic ops, workflow spec, data
lineage. This is what most people mean by "reproducible experiment." - Layer 3: Outputs -> conclusions. Downstream statistical processing,
meta-analysis, plot generation. The layer where analytical
reproducibility (same inputs -> same conclusions under the declared
test) lives.
Compound case the taxonomy must handle: source that compiles to a tool
(Layer 1) that then runs the experiment (Layer 2). Two layers, potentially
different tightness tiers each. Similarly, experiments that consume a
third-party binary the researcher did not build -- Layer 1 is somebody
else's problem, but Layer 2 remains auditable.
Tightness tiers to name explicitly
- Bit-for-bit: byte-equal outputs. Regulatory, forensic,
cryptographic, hermetic-build contexts. - Within tolerance: outputs statistically equivalent under a declared
epsilon or distributional test (KS, chi-square, etc.). The honest bar
for stochastic ML, MCMC, ensemble methods, sampling, GANs. - Runs-to-completion: environment + data available; produces some
result of the declared type. The honest bar for most published
notebooks and the floor darnit should never audit below.
Scope
- Formalize the (layer, tightness) two-axis taxonomy.
- Design how a repo declares its target: candidate surfaces are a new key
in.project/context or a[reproducibility]block in
.baseline.toml. Decide during clarify. - Map darnit-reproducibility's existing controls onto (layer, tightness)
so each control knows which target it audits against.
Non-goals for v0
- The specific signal checks (see #416).
- Auto-remediation of missing tier declarations.
- Rewriting existing controls to be tier-aware in the same PR -- tag
with a default tier and re-tag in a follow-up.
Constitution alignment
- II. Conservative-by-Default: if a repo declares "bit-for-bit" but
a control at that tier is missing evidence, WARN -- never silent PASS. - IV. Never Guess User Values: the (layer, tightness) declaration is
a user-judgment key. darnit MAY propose a tier from repo heuristics
(e.g.,Dockerfilewith digest -> likely bit-for-bit at Layer 1) but
MUST NOT conclude it. - III. TOML-First: tier declaration lives in TOML or
.project/,
not in Python.
Related
- #414 (extractor module).
- #416 (heuristic checks -- each control declares its (layer, tightness)
applicability). - #393 (ontology) -- the taxonomy must serialize into the ontology
vocabulary. - #227 (HermeticBuild) -- the canonical Layer 1 / bit-for-bit control.
- 主要言語
- Python
- スター
- 22
- フォーク
- 27
- 平均マージ
- 1日 23時間
- マージ済み PR（30日）
- 48
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
darnitdevorg/darnit のほかの issue
-
ci needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
darnitdevorg/darnit#607 ·
メンテナーはふだん 2 日以内に返信
-
dependencies module-baseline needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
darnitdevorg/darnit#606 ·
メンテナーはふだん 2 日以内に返信
-
A test starts real Sigstore signing (interactive OIDC login) when the attestation extra is installedオープンmodule-baseline needs triage testing
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
darnitdevorg/darnit#604 ·
メンテナーはふだん 2 日以内に返信
-
`darnit validate` crashes on openssf-baseline.toml summarising levels対応中かも @i-am-paradox が 4 日前に担当しました。 オープンinterface-cli module-core
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
darnitdevorg/darnit#589 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
-
Plugins hard-code version "0.1.0" while the packages are at 0.1.1対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンgood first issue maintenance
難易度 2/5 1〜3時間 初心者へのやさしさ 83/100
darnitdevorg/darnit#587 · コメント 2 件 ·
メンテナーはふだん 2 日以内に返信
darnitdevorg/darnit の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
sierra-research/tau2-bench#635 ·
メンテナーはふだん 1 日以内に返信
-
The add-ssm-model skill requires a Cython simulator; the contributing guide allows Python ones対応中かも @fmuia が今日担当しました。 オープンdocumentation skill
難易度 2/5 1〜3時間 初心者へのやさしさ 83/100
lnccbrown/ssm-simulators#406 ·
メンテナーはふだん 4 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 95/100
btclib-org/btclib-ecc#309 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
jbacalso24/mimry#459 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
[Bug] POST /revoke with a FastMCP access token is a silent no-op (ownership gate never matches)オープンauth bug high-priority
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信