[Feature] Prune manifest partitions before metadata decoding
維護者通常 1 天內回覆
評估
研究方向
The payload names no source files. Start with the existing ManifestEntry and DataFileMeta construction and the bucket-scoped snapshot manifest cache, then read the linked PR #430, which already implements this work. Done means partition predicates run before full entry decoding on ordinary reads, Arrow-cache reads and snapshot-cache hits, with cached values kept complete and existing merge semantics unchanged.
由索引模型根據 Issue 內容生成。
描述
Search before asking
- I searched the existing issues for manifest partition pruning and found no matching issue.
Motivation
Partition-selective scans currently construct complete manifest entries and file metadata before discarding entries outside the requested partition. This also affects repeated reads from the bucket-scoped snapshot manifest cache, which can contain live files from many partitions.
When a query needs only a small subset of those entries, decoding unrelated file metadata and statistics adds avoidable allocation and planning work. The optimization should apply consistently to ordinary manifest reads, manifest Arrow-cache reads, and exact snapshot-cache hits.
Solution
Evaluate partition predicates before constructing full ManifestEntry and DataFileMeta objects, reusing the existing manifest batch and row-range reading paths. On snapshot-cache hits, select the exact retained snapshot and skip other snapshots without materializing their entries.
Preserve the existing lazy-decode option, ADD/DELETE merge semantics, remaining scan filters, retention limits, and metrics. Keep cached values complete across different partition predicates. Predicate evaluation errors must propagate; malformed snapshot-cache values should fall back to rebuilding from manifests. No table or cache format change is required.
Anything else?
Implementation: https://github.com/apache/paimon-cpp/pull/430
Are you willing to submit a PR?
- I'm willing to submit a PR (PR #430 is already open).
- 主要語言
- C++
- 星號
- 68
- 分支
- 34
- 平均合併
- 3 天 18 小時
- 30 天內合併 PR
- 44
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
apache/paimon-cpp 的其他 Issue
-
fix(executor): avoid retaining a callback when lazy worker startup fails可能已有人在做 @realAaronWu 於 3 天前認領。 未關閉
難度 4/5 3-5 天 新手友好度 15/100
apache/paimon-cpp#436 ·
維護者通常 1 天內回覆
-
難度 5/5 一週以上 新手友好度 8/100
apache/paimon-cpp#435 · 1 則留言 ·
維護者通常 1 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 35/100
apache/paimon-cpp#410 ·
維護者通常 1 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 25/100
apache/paimon-cpp#409 ·
維護者通常 1 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 35/100
apache/paimon-cpp#408 ·
維護者通常 1 天內回覆
查看 apache/paimon-cpp 的全部 Issue
相似的 Issue
-
Bug: negative DECIMAL with fewer digits than its scale prints as "0.-5"可能已有人在做 @adsharma 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 82/100
LadybugDB/ladybug#1311 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 1/5 1 小時以內 新手友好度 91/100
FujiNetWIFI/fujinet-firmware#2045 ·
維護者通常 1 天內回覆
-
(S1 - Need confirmation)
難度 2/5 1-3 小時 新手友好度 65/100
CleverRaven/Cataclysm-DDA#89159 ·
維護者通常 1 天內回覆
-
area:libnvmpi bug P3
難度 2/5 1-3 小時 新手友好度 78/100
gjrtimmer/jetson-nvmpi#33 ·
-
bug
難度 2/5 1-3 小時 新手友好度 74/100
維護者通常 1 天內回覆