Large deletes: an optional size limit and chunked deletes
Maintainer antworten meist innerhalb von 6 Tagen
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Anfängerfreundlichkeit
- 18/100
Rechercherichtung
This is a design proposal, not a scoped change: start from the delete() implementation and the cascade preview dj.Diagram.cascade(...).counts() to see how the cascade is enumerated and where the transaction is opened, then read the discussion in #1496 (traversal for delete in 2.4) and #1682 (the confirmation prompt). The open questions — batching by seed-table keys vs per-table rows, and what a partial delete guarantees together with part_integrity — have to be settled with the maintainers before any code is written. Done means an agreed design note covering limit semantics, batching strategy, and resume behaviour, not a merged patch.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Problem
delete() runs the whole cascade in one transaction: it deletes from every affected table, leaves first, and commits at the end. For a large cascade (a subject with many sessions, each with many computed results), that transaction can delete millions of rows. On a server shared with other users it then:
- holds row locks, and on MySQL locks on the index ranges around them, for as long as the deletes take;
- builds a large undo log, so a rollback or a failure late in the cascade is slow;
- can delay replication and backups.
The preview (dj.Diagram.cascade(...).counts()) shows the size beforehand, but nothing stops an unexpectedly large delete from running, and nothing splits a large delete into smaller transactions.
Proposal (to be designed)
- An optional size limit. For example
delete(max_rows=...), or a setting, that refuses the delete before anything is deleted when the cascade would remove more rows than the limit, and reports the per-table counts. A caller who has reviewed the preview raises the limit or passesNone. - Chunked deletes. An option that deletes a large cascade in bounded batches, leaves first, each batch in its own transaction, so locks are held briefly and each transaction stays small.
Questions to settle
- Chunked deletes give up atomicity: a failure partway leaves some rows deleted. What does a partial delete guarantee, and how is it resumed? Deleting leaves first keeps foreign keys valid at every step; part-integrity checks (
part_integrity) then need to hold per batch or be checked up front. - Batches by what: rows of the seed table (each batch is the cascade of a slice of seed keys), or rows per table?
- Interaction with the new traversal for delete in 2.4 (#1496), which decides which rows the cascade reaches.
- Defaults: off by default, with the limit and chunking chosen explicitly by the caller.
Related: #1682 (the confirmation prompt holds the delete transaction open; production code previews and then calls delete(prompt=False)).
- Vorherrschende Sprache
- Python
- Sterne
- 198
- Forks
- 103
- Ø Merge
- 6 T. 1 Std.
- Gemergte PRs (30 T.)
- 10
Entwicklungsumgebung
- Enthält ein Dockerfile oder eine Docker-Compose-Datei
- Keine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus datajoint/datajoint-python
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
datajoint/datajoint-python#1703 ·
Maintainer antworten meist innerhalb von 6 Tagen
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
datajoint/datajoint-python#1702 ·
Maintainer antworten meist innerhalb von 6 Tagen
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
datajoint/datajoint-python#1701 ·
Maintainer antworten meist innerhalb von 6 Tagen
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 74/100
datajoint/datajoint-python#1691 ·
Maintainer antworten meist innerhalb von 6 Tagen
-
dependencies needs-discussion
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
datajoint/datajoint-python#1687 ·
Maintainer antworten meist innerhalb von 6 Tagen
Alle Issues in datajoint/datajoint-python
Ähnliche Issues
-
needs-triage
Schwierigkeit 1/5 1-3 Stunden Anfängerfreundlichkeit 85/100
gke-labs/kube-agents#6620 ·
Maintainer antworten meist innerhalb von 1 Tag
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 75/100
open-webui/open-webui#34719 · 2 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
-
cognition
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
Maintainer antworten meist innerhalb von 1 Tag
-
One non-object JSON record makes the session catalog exitEvtl. vergeben @he-yufeng hat das heute übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 78/100
collabnix/awesome-mcp-lists#433 ·
Maintainer antworten meist innerhalb von 4 Tagen