[Feature] Prune manifest partitions before metadata decoding
维护者通常 1 天内回复
评估
调研方向
The payload names no source files. Start with the existing ManifestEntry and DataFileMeta construction and the bucket-scoped snapshot manifest cache, then read the linked PR #430, which already implements this work. Done means partition predicates run before full entry decoding on ordinary reads, Arrow-cache reads and snapshot-cache hits, with cached values kept complete and existing merge semantics unchanged.
由索引模型根据 Issue 内容生成。
描述
Search before asking
- I searched the existing issues for manifest partition pruning and found no matching issue.
Motivation
Partition-selective scans currently construct complete manifest entries and file metadata before discarding entries outside the requested partition. This also affects repeated reads from the bucket-scoped snapshot manifest cache, which can contain live files from many partitions.
When a query needs only a small subset of those entries, decoding unrelated file metadata and statistics adds avoidable allocation and planning work. The optimization should apply consistently to ordinary manifest reads, manifest Arrow-cache reads, and exact snapshot-cache hits.
Solution
Evaluate partition predicates before constructing full ManifestEntry and DataFileMeta objects, reusing the existing manifest batch and row-range reading paths. On snapshot-cache hits, select the exact retained snapshot and skip other snapshots without materializing their entries.
Preserve the existing lazy-decode option, ADD/DELETE merge semantics, remaining scan filters, retention limits, and metrics. Keep cached values complete across different partition predicates. Predicate evaluation errors must propagate; malformed snapshot-cache values should fall back to rebuilding from manifests. No table or cache format change is required.
Anything else?
Implementation: https://github.com/apache/paimon-cpp/pull/430
Are you willing to submit a PR?
- I'm willing to submit a PR (PR #430 is already open).
- 主要语言
- C++
- 星标
- 68
- 派生
- 34
- 平均合并
- 3 天 18 小时
- 30 天内合并 PR
- 44
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/paimon-cpp 的其他 Issue
-
fix(executor): avoid retaining a callback when lazy worker startup fails可能已有人在做 @realAaronWu 于 3 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 15/100
apache/paimon-cpp#436 ·
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 8/100
apache/paimon-cpp#435 · 1 条评论 ·
维护者通常 1 天内回复
-
enhancement
难度 5/5 一周以上 新手友好度 35/100
apache/paimon-cpp#410 ·
维护者通常 1 天内回复
-
enhancement
难度 5/5 一周以上 新手友好度 25/100
apache/paimon-cpp#409 ·
维护者通常 1 天内回复
-
enhancement
难度 5/5 一周以上 新手友好度 35/100
apache/paimon-cpp#408 ·
维护者通常 1 天内回复
查看 apache/paimon-cpp 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 83/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
-
[BUG] Clipboard History: Buffer over-read in WideCharToMultiByte on double-click paste可能已有人在做 @softmgr 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 75/100
notepad-plus-plus/notepad-plus-plus#18519 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 88/100
justcallmekoko/ESP32Marauder#1635 ·
维护者通常 1 天内回复
-
3.7.6-dev commit 80481ea - files not centered when resolution differs and zoom = "pixel-to-pixel"未关闭
难度 2/5 1-3 小时 新手友好度 78/100
grizzlypeak3d/DJV#931 ·