Make key_source restrict-only: add key_source_restriction, deprecate parent-redefining overrides

Offen
#1,523 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python

Rechercherichtung

Beginne mit der Autopopulate spec §2 und den dort referenzierten How-to-Seiten und verfolge anschließend populate() und make(key), um das aktuelle Verhalten von key_source zu verstehen. Prüfe die koordinierte datajoint-docs-Aufgabe zusammen mit den Anforderungen für den Rollout. Die Aufgabe ist abgeschlossen, wenn key_source_restriction unterstützt wird, Überschreibungen, die das Parent neu definieren, Warnungen ausgeben und die Dokumentation restriction und das master–part-Batching empfiehlt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Summary

Make key_source a restrict-only surface. Today key_source can be overridden freely, including in ways that redefine the parent join — and that is the source of subtle, hard-to-diagnose bugs. Introduce a first-class key_source_restriction that applies a restriction to the default key_source without changing the parents, and steer users (and the docs) toward it.

Background

key_source defaults to the join of the tables referenced by the foreign keys in a computed/imported table's primary key (Autopopulate → Key Source Calculation). populate() iterates over it, passing each key to make(key).

The default is almost always correct. In practice, the only legitimate reason to modify it is to restrict it — compute a subset of the default source (e.g. a completion barrier that fires only when all members of a batch have results). Restricting keeps the parents and the grain intact; each make(key) still receives a fully-formed primary key.

Overriding key_source to redefine the parent join — dropping a parent, adding one, or reconstructing the key from an aggregate — is a different and dangerous operation: it can change or drop primary-key attributes, so make(key) receives a key that is missing attributes or has attributes the table's primary key doesn't declare. The downstream effects (partial inserts, mismatched keys, silent miscomputation) are hard to reason about, and the framework was not designed for it.

Proposal

  1. Introduce key_source_restriction — a declared restriction applied to the default key_source. It may be any valid DataJoint restriction, including a restriction by another table (a dependency visible in key_source but not necessarily as a formal FK edge). The computed source is default_key_source & key_source_restriction; the parents and grain are never changed.
  2. Deprecate user-facing redefinition of key_source. Treat key_source as an internal of the auto-populate machinery. Emit a warning when a subclass overrides key_source in a way that changes the parent join.
  3. Batching is a master–part concern, not a key_source concern. A run/cohort-level batch should be a master keyed on the batch with per-member results on a part — consistent with the DataJoint 2 rule that an auto-populated table introduces no new dimension (only its parts may). This removes the historical "batch by overriding key_source" motivation.

Rollout (per the 2026-07-24 design discussion)

  • 2.4: add key_source_restriction; begin warning on parent-redefining key_source overrides.
  • Docs now: remove key_source-override guidance/recommendations from the documentation; document key_source_restriction as the supported way to narrow a source.
  • Later: make key_source restriction-only (the override path removed as a user surface).

Docs task (coordinated)

datajoint-docs: pull any recommendation to override key_source, and document key_source_restriction + the "batch via master–part" guidance. (Autopopulate spec §2 and the how-to pages.)

cc @MilagrosMarin @ttngu207

Vorherrschende Sprache
Python
Sterne
197
Forks
98
Ø Merge
6 T. 10 Std.
Gemergte PRs (30 T.)
4

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus datajoint/datajoint-python

Alle Issues in datajoint/datajoint-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.