Trustworthy garbage collection: complete reference discovery (#1469) + race-safe deletion (#1445)
還沒有人認領這個 Issue。
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 35/100
- Issue 類型
- 功能
- 描述清晰度
- 基本清楚
- 活躍度
- 活躍
- 技術堆疊
- python
研究方向
先閱讀 issues #1469 和 #1445，以了解所需的實作順序，以及 referenced_paths 和兩階段刪除的要求。接著檢視列出的文件目標，尤其是 reference/specs/garbage-collection.md、how-to/garbage-collection.md 和 reference/specs/codec-api.md。完成的標準是：實作同時涵蓋 discovery 和 race safety，且文件涵蓋 codec contract、工作流程、設定、並行處理以及 restore 語意。
由索引模型根據 Issue 內容生成。
描述
Umbrella tracking the invariant: garbage collection must never delete an object that is still referenced. Today it can, via two independent failure modes that look similar but have different root causes:
- Incomplete discovery — #1469:
dj.gc.scanhardcodes the built-in codec names (hash/blob/attach,object/npy), so files referenced by a custom codec are never enumerated. The store scan then reports live files as orphans andcollect()deletes them. (Row delete also fails to remove custom-codec files.) - Stale discovery / TOCTOU race — #1445: the scan is correct, but an insert during the scan→delete window has its file deleted out from under it.
They are complementary layers of one goal, not duplicates — and must be fixed in order:
- #1469 — complete reference discovery (correctness, FIRST). Codec-owned
referenced_pathshook;scanand delete-cleanup become codec-driven instead of name-hardcoded. Closes active data loss for real pipelines (aeon_mecha). - #1445 — race-safe deletion (hardening, SECOND). Two-phase quarantine → grace → purge with re-check-before-delete;
_trash/prefix for state.
Why the sequence matters
#1445's purge() re-check reruns the scan. On a codec-blind scan (pre-#1469) it would still classify a live custom-codec file as an orphan and purge it after the grace window — so the grace window can't save you from a scan that never sees the reference. Complete discovery (#1469) is the precondition for safe deletion (#1445).
Documentation (datajoint-docs) — to land with the implementation
Add
reference/specs/garbage-collection.md(new normative spec — none exists today). The orphan-determination model, thereferenced_pathscodec contract, the two-phase quarantine/grace/purge state machine, config keys (gc.grace_seconds), re-check/concurrency semantics, backend atomic-move requirements, andrestore. (#1445 explicitly asks for a written spec.)
Update
how-to/garbage-collection.md("Clean Up Object Storage") — document the two-phase workflow (quarantine/purge/restore,grace_seconds); note custom-codec external files are now handled; revise the "single-pass, best-effort" admonition added in #189 once two-phase lands.reference/specs/codec-api.md— documentreferenced_pathsas part of the Codec contract (required for any codec that owns external artifacts).how-to/create-custom-codec.md,explanation/custom-codecs.md,how-to/use-plugin-codecs.md— author guidance: if your codec writes external files, implementreferenced_pathsso delete + GC see them (otherwise files leak or, worse, get misclassified as orphans and deleted).reference/specs/provenance.md— the GC concurrency wording references single-pass semantics; align once two-phase ships (minor).- Optionally a short explainer (e.g. in
explanation/object-storage-overview.md) on how DataJoint tracks external references — codecs own their paths; delete and GC consult them.
- 主要語言
- Python
- 星號
- 197
- 分支
- 98
- 平均合併
- 23 小時 9 分鐘
- 30 天內合併 PR
- 6
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
datajoint/datajoint-python 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 76/100
datajoint/datajoint-python#1539 · 3 則留言 ·
-
Python 3.15 ships Oct 9 and we cap below it; 3.10 went EOL Oct 1可能已有人在做 @dimitri-yatsenko 於 2 天前認領。 未關閉enhancement
datajoint/datajoint-python#1569 · 已指派 1 人 ·
-
bug
難度 3/5 1-2 天 新手友好度 75/100
datajoint/datajoint-python#1564 ·
-
bug
難度 4/5 3-5 天 新手友好度 68/100
datajoint/datajoint-python#1563 ·
-
enhancement
難度 5/5 一週以上 新手友好度 35/100
datajoint/datajoint-python#1562 · 2 則留言 ·
查看 datajoint/datajoint-python 的全部 Issue
相似的 Issue
-
難度 2/5 1-3 小時 新手友好度 70/100
維護者通常 1 天內回覆
-
area:docs
難度 2/5 1-3 小時 新手友好度 72/100
RailtownAI/railtracks#1633 ·
維護者通常 2 天內回覆
-
review-panel severity:low
難度 1/5 1 小時以內 新手友好度 85/100
kristovatlas/coin-accounting#152 ·
維護者通常 1 天內回覆
-
documentation :blue_book:
難度 1/5 1 小時以內 新手友好度 88/100
PennyLaneAI/pennylane#10280 ·
維護者通常 2 天內回覆
-
`pipx reinstall` prints a Python traceback when the reinstall fails可能已有人在做 @ParamTanna 今天認領。 未關閉bug
難度 2/5 1-3 小時 新手友好度 78/100
維護者通常 1 天內回覆