Garbage collection deletes the objects of custom codecs that store through <hash@>
維護者通常 6 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 新手友好度
- 78/100
- Issue 類型
- 缺陷
- 描述清晰度
- 描述清楚
- 活躍度
- 活躍
- 技術堆疊
- mysql, postgresql, python
研究方向
Start with src/datajoint/heading.py and src/datajoint/gc.py, especially the referenced Attribute.is_hash_object and collector scan; read how is_schema_object resolves codec types. Reproduce with the supplied HashChain and BlobChain example, then check that Codec.referenced_paths accounts for all three stored objects. Done means collect(dry_run=False) preserves every row-referenced object and fetching text, obj, and ref still succeeds.
由索引模型根據 Issue 內容生成。
描述
GarbageCollector.collect(dry_run=False) deletes the stored objects of a custom codec that stores through <hash@>, directly or through <blob@> or <attach@>. Fetching those rows then fails with MissingExternalFile.
Attribute.is_hash_object decides by codec name: hash, or blob/attach with a store (heading.py#L182-L198). A custom codec whose get_dtype returns "<hash>" or "<blob>" doesn't match. The collector never scans its column for references (gc.py#L162-L192), so its objects count as orphans. Codec.referenced_paths would report their paths correctly.
This is the part of #1469 that is still open. The fix for #1469 recognizes schema-addressed custom codecs by type, but hash-addressed ones are still recognized by name. It belongs under #1478.
Repro (master, eefe74b4; MySQL and PostgreSQL; a file store local as the default store)
import numpy as np
import datajoint as dj
from datajoint.gc import GarbageCollector
class HashChain(dj.Codec):
name = "hashchain"
def get_dtype(self, is_store):
return "<hash>"
def encode(self, value, *, key=None, context=None, store_name=None):
return value.encode()
def decode(self, stored, *, key=None, context=None):
return stored.decode()
class BlobChain(dj.Codec):
name = "blobchain"
def get_dtype(self, is_store):
return "<blob>"
def encode(self, value, *, key=None, context=None, store_name=None):
return value
def decode(self, stored, *, key=None, context=None):
return stored
@schema
class Note(dj.Manual):
definition = """
note_id : int32
---
text : <hashchain@>
obj : <blobchain@>
ref : <blob@>
"""
Note.insert1(dict(note_id=1, text="hello", obj=np.arange(7), ref=np.arange(7)))
GarbageCollector(schema, store="local").collect()
is_hash_objectis False fortextandobj, and True forref.- All three objects are under
_hash/<schema>/, but the collector finds 1 referenced path and reports thetextandobjobjects as orphaned. collect(dry_run=False)deletes both.Note.fetch1("text")andNote.fetch1("obj")then raiseMissingExternalFile, whilerefstill reads.
Expected
An object referenced by any row is never collected, whatever codec stored it.
Suggested fix
Classify by the resolved codec chain, as is_schema_object does. Resolve the attribute's type with resolve_dtype and test for a HashCodec anywhere in the chain.
- 主要語言
- Python
- 星號
- 198
- 分支
- 103
- 平均合併
- 6 天 1 小時
- 30 天內合併 PR
- 10
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
datajoint/datajoint-python 的其他 Issue
-
bug
難度 2/5 1-3 小時 新手友好度 78/100
datajoint/datajoint-python#1703 ·
維護者通常 6 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 78/100
datajoint/datajoint-python#1702 ·
維護者通常 6 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 78/100
datajoint/datajoint-python#1701 ·
維護者通常 6 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 74/100
datajoint/datajoint-python#1691 ·
維護者通常 6 天內回覆
-
dependencies needs-discussion
難度 2/5 1-3 小時 新手友好度 68/100
datajoint/datajoint-python#1687 ·
維護者通常 6 天內回覆
查看 datajoint/datajoint-python 的全部 Issue
相似的 Issue
-
needs-triage
難度 1/5 1-3 小時 新手友好度 85/100
gke-labs/kube-agents#6620 ·
維護者通常 1 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 75/100
open-webui/open-webui#34719 · 2 則留言 ·
維護者通常 1 天內回覆
-
cognition
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 1 天內回覆
-
One non-object JSON record makes the session catalog exit可能已有人在做 @he-yufeng 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 68/100
維護者通常 1 天內回覆
-
難度 1/5 1 小時以內 新手友好度 78/100
collabnix/awesome-mcp-lists#433 ·
維護者通常 4 天內回覆