Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Trustworthy garbage collection: complete reference discovery (#1469) + race-safe deletion (#1445)

Ouverte
  • Quelque chose ne va pas ?
#1,478 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
35/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
python
Domaine
backend, data-engineering, databases

Piste de recherche

Commencez par lire les issues #1469 et #1445 afin de comprendre la séquence d’implémentation requise ainsi que les exigences liées à referenced_paths et à la suppression en deux phases. Examinez ensuite les cibles de documentation indiquées, en particulier reference/specs/garbage-collection.md, how-to/garbage-collection.md et reference/specs/codec-api.md. C’est terminé lorsque l’implémentation couvre à la fois la découverte et la sécurité contre les conditions de concurrence, et que la documentation couvre le contrat de codec, le workflow, la configuration, la concurrence et la sémantique de restauration.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement

Umbrella tracking the invariant: garbage collection must never delete an object that is still referenced. Today it can, via two independent failure modes that look similar but have different root causes:

  1. Incomplete discovery — #1469: dj.gc.scan hardcodes the built-in codec names (hash/blob/attach, object/npy), so files referenced by a custom codec are never enumerated. The store scan then reports live files as orphans and collect() deletes them. (Row delete also fails to remove custom-codec files.)
  2. Stale discovery / TOCTOU race — #1445: the scan is correct, but an insert during the scan→delete window has its file deleted out from under it.

They are complementary layers of one goal, not duplicates — and must be fixed in order:

  • #1469 — complete reference discovery (correctness, FIRST). Codec-owned referenced_paths hook; scan and delete-cleanup become codec-driven instead of name-hardcoded. Closes active data loss for real pipelines (aeon_mecha).
  • #1445 — race-safe deletion (hardening, SECOND). Two-phase quarantine → grace → purge with re-check-before-delete; _trash/ prefix for state.
Why the sequence matters

#1445's purge() re-check reruns the scan. On a codec-blind scan (pre-#1469) it would still classify a live custom-codec file as an orphan and purge it after the grace window — so the grace window can't save you from a scan that never sees the reference. Complete discovery (#1469) is the precondition for safe deletion (#1445).

Documentation (datajoint-docs) — to land with the implementation

Add

  • reference/specs/garbage-collection.md (new normative spec — none exists today). The orphan-determination model, the referenced_paths codec contract, the two-phase quarantine/grace/purge state machine, config keys (gc.grace_seconds), re-check/concurrency semantics, backend atomic-move requirements, and restore. (#1445 explicitly asks for a written spec.)

Update

  • how-to/garbage-collection.md ("Clean Up Object Storage") — document the two-phase workflow (quarantine / purge / restore, grace_seconds); note custom-codec external files are now handled; revise the "single-pass, best-effort" admonition added in #189 once two-phase lands.
  • reference/specs/codec-api.md — document referenced_paths as part of the Codec contract (required for any codec that owns external artifacts).
  • how-to/create-custom-codec.md, explanation/custom-codecs.md, how-to/use-plugin-codecs.md — author guidance: if your codec writes external files, implement referenced_paths so delete + GC see them (otherwise files leak or, worse, get misclassified as orphans and deleted).
  • reference/specs/provenance.md — the GC concurrency wording references single-pass semantics; align once two-phase ships (minor).
  • Optionally a short explainer (e.g. in explanation/object-storage-overview.md) on how DataJoint tracks external references — codecs own their paths; delete and GC consult them.
Langage dominant
Python
Étoiles
197
Forks
98
Merge moyen
1 j 23 h
PR mergées (30 j)
6

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de datajoint/datajoint-python

Toutes les issues de datajoint/datajoint-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.