Large deletes: an optional size limit and chunked deletes
維護者通常 6 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 18/100
研究方向
This is a design proposal, not a scoped change: start from the delete() implementation and the cascade preview dj.Diagram.cascade(...).counts() to see how the cascade is enumerated and where the transaction is opened, then read the discussion in #1496 (traversal for delete in 2.4) and #1682 (the confirmation prompt). The open questions — batching by seed-table keys vs per-table rows, and what a partial delete guarantees together with part_integrity — have to be settled with the maintainers before any code is written. Done means an agreed design note covering limit semantics, batching strategy, and resume behaviour, not a merged patch.
由索引模型根據 Issue 內容生成。
描述
Problem
delete() runs the whole cascade in one transaction: it deletes from every affected table, leaves first, and commits at the end. For a large cascade (a subject with many sessions, each with many computed results), that transaction can delete millions of rows. On a server shared with other users it then:
- holds row locks, and on MySQL locks on the index ranges around them, for as long as the deletes take;
- builds a large undo log, so a rollback or a failure late in the cascade is slow;
- can delay replication and backups.
The preview (dj.Diagram.cascade(...).counts()) shows the size beforehand, but nothing stops an unexpectedly large delete from running, and nothing splits a large delete into smaller transactions.
Proposal (to be designed)
- An optional size limit. For example
delete(max_rows=...), or a setting, that refuses the delete before anything is deleted when the cascade would remove more rows than the limit, and reports the per-table counts. A caller who has reviewed the preview raises the limit or passesNone. - Chunked deletes. An option that deletes a large cascade in bounded batches, leaves first, each batch in its own transaction, so locks are held briefly and each transaction stays small.
Questions to settle
- Chunked deletes give up atomicity: a failure partway leaves some rows deleted. What does a partial delete guarantee, and how is it resumed? Deleting leaves first keeps foreign keys valid at every step; part-integrity checks (
part_integrity) then need to hold per batch or be checked up front. - Batches by what: rows of the seed table (each batch is the cascade of a slice of seed keys), or rows per table?
- Interaction with the new traversal for delete in 2.4 (#1496), which decides which rows the cascade reaches.
- Defaults: off by default, with the limit and chunking chosen explicitly by the caller.
Related: #1682 (the confirmation prompt holds the delete transaction open; production code previews and then calls delete(prompt=False)).
- 主要語言
- Python
- 星號
- 198
- 分支
- 103
- 平均合併
- 6 天 1 小時
- 30 天內合併 PR
- 10
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
datajoint/datajoint-python 的其他 Issue
-
bug
難度 2/5 1-3 小時 新手友好度 78/100
datajoint/datajoint-python#1703 ·
維護者通常 6 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 78/100
datajoint/datajoint-python#1702 ·
維護者通常 6 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 78/100
datajoint/datajoint-python#1701 ·
維護者通常 6 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 74/100
datajoint/datajoint-python#1691 ·
維護者通常 6 天內回覆
-
dependencies needs-discussion
難度 2/5 1-3 小時 新手友好度 68/100
datajoint/datajoint-python#1687 ·
維護者通常 6 天內回覆
查看 datajoint/datajoint-python 的全部 Issue
相似的 Issue
-
needs-triage
難度 1/5 1-3 小時 新手友好度 85/100
gke-labs/kube-agents#6620 ·
維護者通常 1 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 75/100
open-webui/open-webui#34719 · 2 則留言 ·
維護者通常 1 天內回覆
-
cognition
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 1 天內回覆
-
One non-object JSON record makes the session catalog exit可能已有人在做 @he-yufeng 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 68/100
維護者通常 1 天內回覆
-
難度 1/5 1 小時以內 新手友好度 78/100
collabnix/awesome-mcp-lists#433 ·
維護者通常 4 天內回覆