Add maintenance action to remove dangling delete files
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- aws, python
- Domain
- data-engineering, databases
Research direction
Start with MaintenanceTable and ManifestWriterV2, then inspect the referenced write_manifest_list and snapshot update internals. Use the stated per-partition sequence rules and classification test matrix as acceptance criteria; done means a metadata-only replace commit removes only dangling deletes while preserving ambiguous and live entries.
Written by the indexing model from the issue text.
Description
Feature Request / Improvement
Streaming upsert writers (e.g. AWS Firehose Iceberg delivery) write equality-delete files on every commit. Compaction applies them into rewritten data files but leaves the entries in the manifests, and expire_snapshots can't touch files the current snapshot still references — so they accumulate without bound. On one of our production tables we measured ~90K dangling delete entries growing ~4.6K/day, and every query planning over recent partitions has to read the ever-growing delete manifests.
Java Iceberg handles this (rewrite_data_files with remove-dangling-deletes, RemoveDanglingDeletesSparkAction), but PyIceberg's MaintenanceTable currently only has expire_snapshots, and engines like Athena expose no statement for it either — so users on Athena/Firehose stacks have no non-Spark way out.
Proposal: table.maintenance.remove_dangling_deletes() — a metadata-only commit that:
- classifies per
(partition_spec_id, partition): an equality delete at sequence s is dangling iff no live data file in that partition has sequence < s (position deletes: <= s); ambiguous cases (unpartitioned specs, unknown content) are kept - carries data manifests through unchanged, drops fully-dangling delete manifests, rewrites mixed ones to their surviving entries, and commits as a
replacesnapshot against the current ref
One enabler is worth a small standalone fix first: ManifestWriterV2 hardcodes content=data, so PyIceberg currently can't write delete-content manifests at all.
We have a working implementation built on PyIceberg 0.12 internals (write_manifest_list, a ManifestWriterV2 subclass, AddSnapshotUpdate/SetSnapshotRefUpdate with AssertRefSnapshotId), validated against production Glue/Athena tables, with a test matrix for the classification rules. Happy to contribute it if there's interest.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 589
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 72
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3996 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#3979 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg-python#3866 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3836 · 1 comment ·
All issues in apache/iceberg-python
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
learningequality/ricecooker#747 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
run-llama/llama_index#23199 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
KhronosGroup/glTF-Blender-IO#2769 ·