Add maintenance action to remove dangling delete files
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 45/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- aws, python
调研方向
从 MaintenanceTable 和 ManifestWriterV2 开始,然后检查所引用的 write_manifest_list 和 snapshot 更新内部实现。使用所述的按分区序列规则和分类测试矩阵作为验收标准;完成意味着纯元数据替换提交只移除悬空删除,同时保留有歧义的条目和活动条目。
由索引模型根据 Issue 内容生成。
描述
Feature Request / Improvement
Streaming upsert writers (e.g. AWS Firehose Iceberg delivery) write equality-delete files on every commit. Compaction applies them into rewritten data files but leaves the entries in the manifests, and expire_snapshots can't touch files the current snapshot still references — so they accumulate without bound. On one of our production tables we measured ~90K dangling delete entries growing ~4.6K/day, and every query planning over recent partitions has to read the ever-growing delete manifests.
Java Iceberg handles this (rewrite_data_files with remove-dangling-deletes, RemoveDanglingDeletesSparkAction), but PyIceberg's MaintenanceTable currently only has expire_snapshots, and engines like Athena expose no statement for it either — so users on Athena/Firehose stacks have no non-Spark way out.
Proposal: table.maintenance.remove_dangling_deletes() — a metadata-only commit that:
- classifies per
(partition_spec_id, partition): an equality delete at sequence s is dangling iff no live data file in that partition has sequence < s (position deletes: <= s); ambiguous cases (unpartitioned specs, unknown content) are kept - carries data manifests through unchanged, drops fully-dangling delete manifests, rewrites mixed ones to their surviving entries, and commits as a
replacesnapshot against the current ref
One enabler is worth a small standalone fix first: ManifestWriterV2 hardcodes content=data, so PyIceberg currently can't write delete-content manifests at all.
We have a working implementation built on PyIceberg 0.12 internals (write_manifest_list, a ManifestWriterV2 subclass, AddSnapshotUpdate/SetSnapshotRefUpdate with AssertRefSnapshotId), validated against production Glue/Athena tables, with a test matrix for the classification rules. Happy to contribute it if there's interest.
- 主要语言
- Python
- 星标
- 1.1k
- 派生
- 589
- 平均合并
- 1 天 20 小时
- 30 天内合并 PR
- 68
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/iceberg-python 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
apache/iceberg-python#4010 · 1 个 reaction ·
-
kind:bug
难度 1/5 1 小时以内 新手友好度 92/100
apache/iceberg-python#4006 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
apache/iceberg-python#3996 ·
-
bug
难度 2/5 1-3 小时 新手友好度 72/100
apache/iceberg-python#3979 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
apache/iceberg-python#3885 ·
查看 apache/iceberg-python 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 75/100
stephrobert/dsoxlab#238 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 75/100
sublimehq/package_control#1780 ·
-
难度 2/5 1-3 小时 新手友好度 65/100
-
难度 2/5 1-3 小时 新手友好度 70/100
nwg-piotr/nwg-displays#145 ·