Garbage collection deletes the objects of custom codecs that store through <hash@>

Aberta
#1,700 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 6 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
78/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
mysql, postgresql, python

Direção de pesquisa

Start with src/datajoint/heading.py and src/datajoint/gc.py, especially the referenced Attribute.is_hash_object and collector scan; read how is_schema_object resolves codec types. Reproduce with the supplied HashChain and BlobChain example, then check that Codec.referenced_paths accounts for all three stored objects. Done means collect(dry_run=False) preserves every row-referenced object and fetching text, obj, and ref still succeeds.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug

GarbageCollector.collect(dry_run=False) deletes the stored objects of a custom codec that stores through <hash@>, directly or through <blob@> or <attach@>. Fetching those rows then fails with MissingExternalFile.

Attribute.is_hash_object decides by codec name: hash, or blob/attach with a store (heading.py#L182-L198). A custom codec whose get_dtype returns "<hash>" or "<blob>" doesn't match. The collector never scans its column for references (gc.py#L162-L192), so its objects count as orphans. Codec.referenced_paths would report their paths correctly.

This is the part of #1469 that is still open. The fix for #1469 recognizes schema-addressed custom codecs by type, but hash-addressed ones are still recognized by name. It belongs under #1478.

Repro (master, eefe74b4; MySQL and PostgreSQL; a file store local as the default store)

import numpy as np
import datajoint as dj
from datajoint.gc import GarbageCollector


class HashChain(dj.Codec):
    name = "hashchain"

    def get_dtype(self, is_store):
        return "<hash>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value.encode()

    def decode(self, stored, *, key=None, context=None):
        return stored.decode()


class BlobChain(dj.Codec):
    name = "blobchain"

    def get_dtype(self, is_store):
        return "<blob>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value

    def decode(self, stored, *, key=None, context=None):
        return stored


@schema
class Note(dj.Manual):
    definition = """
    note_id : int32
    ---
    text : <hashchain@>
    obj : <blobchain@>
    ref : <blob@>
    """


Note.insert1(dict(note_id=1, text="hello", obj=np.arange(7), ref=np.arange(7)))
GarbageCollector(schema, store="local").collect()
  • is_hash_object is False for text and obj, and True for ref.
  • All three objects are under _hash/<schema>/, but the collector finds 1 referenced path and reports the text and obj objects as orphaned.
  • collect(dry_run=False) deletes both.
  • Note.fetch1("text") and Note.fetch1("obj") then raise MissingExternalFile, while ref still reads.

Expected

An object referenced by any row is never collected, whatever codec stored it.

Suggested fix

Classify by the resolved codec chain, as is_schema_object does. Resolve the attribute's type with resolve_dtype and test for a HashCodec anywhere in the chain.

Linguagem predominante
Python
Estrelas
198
Forks
103
Merge médio
6d 1h
PRs com merge (30d)
10

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de datajoint/datajoint-python

Todas as issues de datajoint/datajoint-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.