datalake_fdw: merge-on-read and DML
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start with parent issue #2008 and the prerequisite work C and D, then trace WrapPositionDeleteFilter on the hidden row-ordinal column and C's writer. Done means cross-system deletes are visible, UPDATE/DELETE counts and rollback behavior are correct, concurrent commits do not lose updates, and scans remain within the tracked memory budget.
Written by the indexing model from the issue text.
Description
Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is E.
Scope
- Merge-on-read:
WrapPositionDeleteFilterimplemented on the hidden row-ordinal column; position deletes applied while scanning, equality deletes after. DELETEandUPDATE: rows identified by (file, position) through the scan, written as position delete files;UPDATEis delete plus insert through C's writer; one snapshot per statement.- A commit that loses the race is retried or fails with a serialization error, never silently dropped.
VACUUMstays the current no-op; compaction and snapshot expiry are a follow-up issue.
Out of scope
Copy-on-write, partition evolution, branches and tags (#1683 §2.3).
Depends on
C, D.
Acceptance
- Deletes made in Cloudberry are seen by Spark and the reverse.
UPDATE/DELETEcounts match; a rolled-back statement changes nothing visible.- Two concurrent transactions on one table: one commits, the other fails or retries; no lost update.
- A scan over many small delete files stays within the tracked memory budget.
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 248
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 38
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/cloudberry
-
type: Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/cloudberry#1885 · 2 reactions ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
apache/cloudberry#1825 ·
-
type: Bug
Difficulty 4/5 3-5 days Newbie friendliness 52/100
apache/cloudberry#2026 · 1 comment · 1 reaction ·
-
type: Bug
Difficulty 4/5 3-5 days Newbie friendliness 48/100
apache/cloudberry#2025 · 1 reaction ·
-
datalake
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/cloudberry#2019 ·
All issues in apache/cloudberry
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
duckdb/duckdb-wasm#2258 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
rofl0r/microsocks#110 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
espressif/esp-aws-iot#261 ·
-
os:linux
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
mpv-player/mpv#18510 · 2 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100