Garbage collection deletes the objects of custom codecs that store through <hash@>

Abierto
#1,700 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 6 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
mysql, postgresql, python

Línea de trabajo

Start with src/datajoint/heading.py and src/datajoint/gc.py, especially the referenced Attribute.is_hash_object and collector scan; read how is_schema_object resolves codec types. Reproduce with the supplied HashChain and BlobChain example, then check that Codec.referenced_paths accounts for all three stored objects. Done means collect(dry_run=False) preserves every row-referenced object and fetching text, obj, and ref still succeeds.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

GarbageCollector.collect(dry_run=False) deletes the stored objects of a custom codec that stores through <hash@>, directly or through <blob@> or <attach@>. Fetching those rows then fails with MissingExternalFile.

Attribute.is_hash_object decides by codec name: hash, or blob/attach with a store (heading.py#L182-L198). A custom codec whose get_dtype returns "<hash>" or "<blob>" doesn't match. The collector never scans its column for references (gc.py#L162-L192), so its objects count as orphans. Codec.referenced_paths would report their paths correctly.

This is the part of #1469 that is still open. The fix for #1469 recognizes schema-addressed custom codecs by type, but hash-addressed ones are still recognized by name. It belongs under #1478.

Repro (master, eefe74b4; MySQL and PostgreSQL; a file store local as the default store)

import numpy as np
import datajoint as dj
from datajoint.gc import GarbageCollector


class HashChain(dj.Codec):
    name = "hashchain"

    def get_dtype(self, is_store):
        return "<hash>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value.encode()

    def decode(self, stored, *, key=None, context=None):
        return stored.decode()


class BlobChain(dj.Codec):
    name = "blobchain"

    def get_dtype(self, is_store):
        return "<blob>"

    def encode(self, value, *, key=None, context=None, store_name=None):
        return value

    def decode(self, stored, *, key=None, context=None):
        return stored


@schema
class Note(dj.Manual):
    definition = """
    note_id : int32
    ---
    text : <hashchain@>
    obj : <blobchain@>
    ref : <blob@>
    """


Note.insert1(dict(note_id=1, text="hello", obj=np.arange(7), ref=np.arange(7)))
GarbageCollector(schema, store="local").collect()
  • is_hash_object is False for text and obj, and True for ref.
  • All three objects are under _hash/<schema>/, but the collector finds 1 referenced path and reports the text and obj objects as orphaned.
  • collect(dry_run=False) deletes both.
  • Note.fetch1("text") and Note.fetch1("obj") then raise MissingExternalFile, while ref still reads.

Expected

An object referenced by any row is never collected, whatever codec stored it.

Suggested fix

Classify by the resolved codec chain, as is_schema_object does. Resolve the attribute's type with resolve_dtype and test for a HashCodec anywhere in the chain.

Lenguaje dominante
Python
Estrellas
198
Forks
103
Merge medio
6 d 1 h
PR fusionados (30 d)
10

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de datajoint/datajoint-python

Todos los issues de datajoint/datajoint-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.