Reproducibility Review (main)

Bắt đầu với README gốc và README của campaign, sau đó kiểm tra fpdesign/campaign.py quanh build_argparser và các bước kiểm tra output đã được cache của nó. Xem lại notebook in-silico-test được đề cập, README của GFP_DMS, environment.yml và các tham chiếu đến đường dẫn đã cũ để phân biệt các thay đổi sửa tài liệu với các vấn đề về khả năng tái lập. Được xem là hoàn tất khi các lệnh và đường dẫn khôi phục được ghi trong tài liệu mô tả chính xác hành vi của một fresh clone và tránh traceback đã báo cáo hoặc các tham chiếu gây hiểu nhầm.

Repository: Arcadia-Science/FPguide
Branch: main
Commit: 29935e8cb7
Reviewed: 2026-09-03 21:27 UTC

Reproducibility review — Arcadia-Science/FPguide

Summary

This repository is unusually well documented for reproducibility: every folder carries its own README with ordered commands, the root README maps each entry point onto a four-tier "what can you actually regenerate" table, the conda environment is fully pinned with lock files plus OS/driver notes, and the tracked artifacts were deliberately chosen so the cheap entry points (msa_conservation/, experiment/, Section 1 of GFP_DMS/visualization.ipynb) run from a bare clone. Someone could clone this today and, after one ~325 MB release download, re-run the GFP_DMS analysis and the EGFP campaign analysis as documented. The remaining friction is at the heavy end: the in-silico-test/ deployed checkpoints are not published and their refit is not deterministic, and the committed design CSVs make the documented campaign "run" commands no-ops on a fresh clone.

Critical issues

None found.

Warnings

• The in-silico-test/ deployed checkpoints are not published, and refitting them is not bit-reproducible. design_common.SURR_CKPT (trained_models/surrogate_final/cnn-max-d1_trainval.pt, design_common.py:147) and ORAC_CKPT (trained_models/oracle_sweep/cnn-max-d1_s0.pt, line 148) fall under the repo-wide *.pt ignore, as does trained_models/surrogate_cv3.csv, from which train_final_surrogate.py:40 derives its epoch budget. The rebuild path is documented (root README Tier 3, in-silico-test/README.md pipeline), but it is the full chain — ~2 GB of ESM-2/ProstT5 caches, sweep_peak_oracle.py --role both, cv_all_surrogate.py, train_final_surrogate.py — and train_final_surrogate.py:52 sets only torch.manual_seed(0) with no cuDNN determinism pinned, so a reproducer's surrogate will differ slightly from the one behind the numbers in in-silico-test/README.md ("Design results"). Either ship these two checkpoints alongside the reference clouds, or state in the README that exact numeric agreement is not expected.
• The documented campaign run commands silently do nothing on a fresh clone. The design CSVs under design-campaign-EGFP/*/designs*/ are committed, and fpdesign/campaign.py:670–690 skips any pair/cell whose CSV already covers --trials. build_argparser (campaign.py:182–214) exposes no --force — only --rescore, --backfill-ppl, --probe — so bash run_sweep.sh / bash run_campaign.sh from the campaign README's "Run" section print "cached" and exit. Add a line telling reproducers to pass a different --outdir (or delete the CSVs) to actually re-execute. in-silico-test's equivalent _complete() skip is documented under "Notes and gotchas"; the campaign's is not.

Polish/nice to have

• in-silico-test/sweep_results.ipynb cell 6 does a bare pd.read_csv("trained_models/surrogate_cv3.csv") and raises FileNotFoundError on a clone, whereas the root README's Tier 3 table says those entry points "fail with the command to run, not a traceback". load_sweep in the same notebook already degrades gracefully; the CV cell could too.
• Paths with spaces and parentheses — experiment/plasmid constructs/ and (bright)his-morange_msa_07-in-pet-28a.fasta — force quoting and URL-escaping (experiment/README.md already has to write plasmid%20constructs).
• ~175 MB of RCSB PDBx cache is committed under structures/experimental/ even though pockets.py self-populates it on a miss (structures/README.md says as much). 1G7K.pdbx and 1GFL.pdbx additionally exist at both structures/ and structures/experimental/, though that duplication is functional (pockets.STRUCT reads the top level, experimental_window the subdirectory).
• The root README's Environment section documents only conda env create -f environment.yml; the exact-restore path (environment.linux-64.lock plus requirements.linux-64.txt plus pip install -e ./fpbase-extractor) is only in comments inside environment.yml. Worth surfacing.
• Stale internal references: GFP_DMS/brightness_threshold.py:16 and GFP_DMS/embed_dms.py:16 name transform_dms.py, now transform_avgfp_dms.py; the stage bootstraps in in-silico-test/1_surrogate_oracle_training/*.py and 2_design_task_specification/*.py still prepend a removed msa/ directory to sys.path (harmless, but misleading).
• GFP_DMS/README.md lane A ends in jupyter nbconvert --to notebook --execute --inplace visualization.ipynb, which rewrites a tracked 268 KB notebook and dirties git status on every reproduction run; --output to a scratch path would avoid that.

Suggestions

• Publish the in-silico-test/ surrogate and oracle checkpoints (and surrogate_cv3.csv) in the same release as reference-cloud-v1. They are small, and this is the single change that moves the pub's core design experiment from "re-derive over many GPU-hours, non-deterministically" to "download and re-run".
• Give the design engines an explicit --force, or have the wrapper scripts default to a timestamped --outdir, so "re-run the campaign" is one documented command rather than a manual delete of committed files.

Automated reproducibility review • model: claude-opus-5 • tokens: 569,237 in / 6,953 out • cache: 569,235 written, 0 read