`darnit-reproducibility`: reproducibility layer + tightness taxonomy
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start by comparing the taxonomy needs in this issue with #393 (ontology) and #227 (HermeticBuild), then review the declaration options: .project/ context and [reproducibility] in .baseline.toml. Clarify the declaration format and how existing controls map to layer and tightness; done means the taxonomy handles compound layers and the third-party-binary case and can serialize into the ontology vocabulary.
由索引模型根据 Issue 内容生成。
描述
Summary
"Reproducible" is not a single concept. Auditing meaningfully requires two
independent axes:
- Layer being reproduced.
- Tightness of match at that layer.
Without this taxonomy, users cannot declare what they are aiming for and
darnit cannot audit meaningfully -- the required signals differ per (layer,
tightness) pair. A repo can be statistically reproducible without pinning
PYTHONHASHSEED; a regulatory-grade forensic tool cannot. Conflating these
under one "reproducibility" label produces false PASS or false FAIL,
depending on which direction the assumption drifts.
Layers to name explicitly
- Layer 1: Source -> built tool. Deterministic compilation,
SOURCE_DATE_EPOCH, pinned toolchain, container-build digest. This is
build reproducibility; overlaps withRE-02.01 HermeticBuild(#227). - Layer 2: Built tool + inputs -> outputs. Pinned execution
environment, seeded RNG, deterministic ops, workflow spec, data
lineage. This is what most people mean by "reproducible experiment." - Layer 3: Outputs -> conclusions. Downstream statistical processing,
meta-analysis, plot generation. The layer where analytical
reproducibility (same inputs -> same conclusions under the declared
test) lives.
Compound case the taxonomy must handle: source that compiles to a tool
(Layer 1) that then runs the experiment (Layer 2). Two layers, potentially
different tightness tiers each. Similarly, experiments that consume a
third-party binary the researcher did not build -- Layer 1 is somebody
else's problem, but Layer 2 remains auditable.
Tightness tiers to name explicitly
- Bit-for-bit: byte-equal outputs. Regulatory, forensic,
cryptographic, hermetic-build contexts. - Within tolerance: outputs statistically equivalent under a declared
epsilon or distributional test (KS, chi-square, etc.). The honest bar
for stochastic ML, MCMC, ensemble methods, sampling, GANs. - Runs-to-completion: environment + data available; produces some
result of the declared type. The honest bar for most published
notebooks and the floor darnit should never audit below.
Scope
- Formalize the (layer, tightness) two-axis taxonomy.
- Design how a repo declares its target: candidate surfaces are a new key
in.project/context or a[reproducibility]block in
.baseline.toml. Decide during clarify. - Map darnit-reproducibility's existing controls onto (layer, tightness)
so each control knows which target it audits against.
Non-goals for v0
- The specific signal checks (see #416).
- Auto-remediation of missing tier declarations.
- Rewriting existing controls to be tier-aware in the same PR -- tag
with a default tier and re-tag in a follow-up.
Constitution alignment
- II. Conservative-by-Default: if a repo declares "bit-for-bit" but
a control at that tier is missing evidence, WARN -- never silent PASS. - IV. Never Guess User Values: the (layer, tightness) declaration is
a user-judgment key. darnit MAY propose a tier from repo heuristics
(e.g.,Dockerfilewith digest -> likely bit-for-bit at Layer 1) but
MUST NOT conclude it. - III. TOML-First: tier declaration lives in TOML or
.project/,
not in Python.
Related
- #414 (extractor module).
- #416 (heuristic checks -- each control declares its (layer, tightness)
applicability). - #393 (ontology) -- the taxonomy must serialize into the ontology
vocabulary. - #227 (HermeticBuild) -- the canonical Layer 1 / bit-for-bit control.
- 主要语言
- Python
- 星标
- 22
- 派生
- 27
- 平均合并
- 1 天 20 小时
- 30 天内合并 PR
- 39
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
darnitdevorg/darnit 的其他 Issue
-
`darnit validate` crashes on openssf-baseline.toml summarising levels可能已有人在做 @i-am-paradox 于 2 天前认领。 未关闭needs triage
难度 2/5 1-3 小时 新手友好度 78/100
darnitdevorg/darnit#589 ·
维护者通常 1 天内回复
-
Plugins hard-code version "0.1.0" while the packages are at 0.1.1可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭needs triage
难度 2/5 1-3 小时 新手友好度 83/100
darnitdevorg/darnit#587 ·
维护者通常 1 天内回复
-
needs triage
难度 2/5 1-3 小时 新手友好度 65/100
darnitdevorg/darnit#585 ·
维护者通常 1 天内回复
-
quality
难度 2/5 1-3 小时 新手友好度 66/100
darnitdevorg/darnit#523 ·
维护者通常 1 天内回复
-
framework-design.md CEL examples use '!= null', which cel-python rejects可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭documentation good first issue
难度 2/5 1-3 小时 新手友好度 76/100
darnitdevorg/darnit#516 ·
维护者通常 1 天内回复
查看 darnitdevorg/darnit 的全部 Issue
相似的 Issue
-
python triage
难度 2/5 1-3 小时 新手友好度 72/100
microsoft/semantic-kernel#14615 ·
维护者通常 3 天内回复
-
难度 2/5 1-3 小时 新手友好度 65/100
-
bug
难度 2/5 1-3 小时 新手友好度 74/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
shackett/shackett-utils#19 ·
-
难度 2/5 1-3 小时 新手友好度 72/100
UKGovernmentBEIS/control-arena#917 ·
维护者通常 1 天内回复