Garbage collection deletes the objects of custom codecs that store through <hash@>

Aperta
#1,700 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
mysql, postgresql, python

Direzione di ricerca

Start with src/datajoint/heading.py and src/datajoint/gc.py, especially the referenced Attribute.is_hash_object and collector scan; read how is_schema_object resolves codec types. Reproduce with the supplied HashChain and BlobChain example, then check that Codec.referenced_paths accounts for all three stored objects. Done means collect(dry_run=False) preserves every row-referenced object and fetching text, obj, and ref still succeeds.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

GarbageCollector.collect(dry_run=False) deletes the stored objects of a custom codec that stores through <hash@>, directly or through <blob@> or <attach@>. Fetching those rows then fails with MissingExternalFile.

Attribute.is_hash_object decides by codec name: hash, or blob/attach with a store (heading.py#L182-L198). A custom codec whose get_dtype returns "<hash>" or "<blob>" doesn't match. The collector never scans its column for references (gc.py#L162-L192), so its objects count as orphans. Codec.referenced_paths would report their paths correctly.

This is the part of #1469 that is still open. The fix for #1469 recognizes schema-addressed custom codecs by type, but hash-addressed ones are still recognized by name. It belongs under #1478.

Repro (master, eefe74b4; MySQL and PostgreSQL; a file store local as the default store)

import numpy as np
import datajoint as dj
from datajoint.gc import GarbageCollector


class HashChain(dj.Codec):
    name = "hashchain"

    def get_dtype(self, is_store):
        return "<hash>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value.encode()

    def decode(self, stored, *, key=None, context=None):
        return stored.decode()


class BlobChain(dj.Codec):
    name = "blobchain"

    def get_dtype(self, is_store):
        return "<blob>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value

    def decode(self, stored, *, key=None, context=None):
        return stored


@schema
class Note(dj.Manual):
    definition = """
    note_id : int32
    ---
    text : <hashchain@>
    obj : <blobchain@>
    ref : <blob@>
    """


Note.insert1(dict(note_id=1, text="hello", obj=np.arange(7), ref=np.arange(7)))
GarbageCollector(schema, store="local").collect()
  • is_hash_object is False for text and obj, and True for ref.
  • All three objects are under _hash/<schema>/, but the collector finds 1 referenced path and reports the text and obj objects as orphaned.
  • collect(dry_run=False) deletes both.
  • Note.fetch1("text") and Note.fetch1("obj") then raise MissingExternalFile, while ref still reads.

Expected

An object referenced by any row is never collected, whatever codec stored it.

Suggested fix

Classify by the resolved codec chain, as is_schema_object does. Resolve the attribute's type with resolve_dtype and test for a HashCodec anywhere in the chain.

Lingua principale
Python
Stelle
198
Fork
103
Merge medio
6g 1h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di datajoint/datajoint-python

Tutte le issue di datajoint/datajoint-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.