Garbage collection deletes the objects of custom codecs that store through <hash@>

未關閉
#1,700 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 6 天內回覆

還沒有人認領這個 Issue。

評估

難度
3/5
預估耗時
1-2 天
新手友好度
78/100
Issue 類型
缺陷
描述清晰度
描述清楚
活躍度
活躍
技術堆疊
mysql, postgresql, python

研究方向

Start with src/datajoint/heading.py and src/datajoint/gc.py, especially the referenced Attribute.is_hash_object and collector scan; read how is_schema_object resolves codec types. Reproduce with the supplied HashChain and BlobChain example, then check that Codec.referenced_paths accounts for all three stored objects. Done means collect(dry_run=False) preserves every row-referenced object and fetching text, obj, and ref still succeeds.

由索引模型根據 Issue 內容生成。

描述

bug

GarbageCollector.collect(dry_run=False) deletes the stored objects of a custom codec that stores through <hash@>, directly or through <blob@> or <attach@>. Fetching those rows then fails with MissingExternalFile.

Attribute.is_hash_object decides by codec name: hash, or blob/attach with a store (heading.py#L182-L198). A custom codec whose get_dtype returns "<hash>" or "<blob>" doesn't match. The collector never scans its column for references (gc.py#L162-L192), so its objects count as orphans. Codec.referenced_paths would report their paths correctly.

This is the part of #1469 that is still open. The fix for #1469 recognizes schema-addressed custom codecs by type, but hash-addressed ones are still recognized by name. It belongs under #1478.

Repro (master, eefe74b4; MySQL and PostgreSQL; a file store local as the default store)

import numpy as np
import datajoint as dj
from datajoint.gc import GarbageCollector


class HashChain(dj.Codec):
    name = "hashchain"

    def get_dtype(self, is_store):
        return "<hash>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value.encode()

    def decode(self, stored, *, key=None, context=None):
        return stored.decode()


class BlobChain(dj.Codec):
    name = "blobchain"

    def get_dtype(self, is_store):
        return "<blob>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value

    def decode(self, stored, *, key=None, context=None):
        return stored


@schema
class Note(dj.Manual):
    definition = """
    note_id : int32
    ---
    text : <hashchain@>
    obj : <blobchain@>
    ref : <blob@>
    """


Note.insert1(dict(note_id=1, text="hello", obj=np.arange(7), ref=np.arange(7)))
GarbageCollector(schema, store="local").collect()
  • is_hash_object is False for text and obj, and True for ref.
  • All three objects are under _hash/<schema>/, but the collector finds 1 referenced path and reports the text and obj objects as orphaned.
  • collect(dry_run=False) deletes both.
  • Note.fetch1("text") and Note.fetch1("obj") then raise MissingExternalFile, while ref still reads.

Expected

An object referenced by any row is never collected, whatever codec stored it.

Suggested fix

Classify by the resolved codec chain, as is_schema_object does. Resolve the attribute's type with resolve_dtype and test for a HashCodec anywhere in the chain.

主要語言
Python
星號
198
分支
103
平均合併
6 天 1 小時
30 天內合併 PR
10

環境準備

  • 提供 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

datajoint/datajoint-python 的其他 Issue

查看 datajoint/datajoint-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。