[Feature] Unpack predicate boolean kernel bitmaps a byte at a time
还没有人认领这个 Issue。
评估
调研方向
从 src/paimon/common/utils/arrow/ 开始,找到 MultiLiteralsLeafFunction 和 NullFalseLeafBinaryFunction,然后通过 LeafFunction::Test 跟踪它们现有的 bitmap-to-byte 循环。添加共享 helper 和一个 property test,覆盖每个 offset 和 length、两种 negate 值,以及未按字节对齐的 validity 区间。当 helper 与逐行 accessor reference 匹配,并且两个 caller 都返回未改变的 bytes 时,即表示完成。
由索引模型根据 Issue 内容生成。
描述
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Two predicate leaf functions evaluate a batch by running an arrow::compute kernel and reading back the arrow::BooleanArray it writes: MultiLiteralsLeafFunction (IN / NOT IN, via IsIn) and NullFalseLeafBinaryFunction (the comparison functions). A kernel returns a bitmap, one bit per row, but LeafFunction::Test returns std::vector<char>, one byte per row, so both call sites spread the bits over bytes with the same per-row loop: test IsNull, read Value, apply the negation NOT IN needs, store a byte. That is a shift, a mask and a byte store per row, duplicated across the two call sites, on the selection path every filtered batch goes through.
Solution
Extract the spread into one helper, ArrowUtils::UnpackBooleansToBytes(array, negate), and read the bitmap a byte at a time instead of a bit at a time:
- A compile-time table maps each of the 256 bitmap bytes to the eight bytes it expands to, so the aligned body produces eight rows per iteration with one lookup and one 8-byte store.
- A scalar head and tail cover the rows sharing a partial leading or trailing byte, which is where the array offset is not byte-aligned; a batch a kernel has just written is aligned, so it takes the fast body throughout.
- The offset and the validity bitmap are honoured exactly as
BooleanArray::Value()andArray::IsValid()honour them, and a null row unpacks to 0 whatever the value bitmap holds for it, which is what bothIN/NOT INand everyNullFalseLeafBinaryFunctionrequire.
MultiLiteralsLeafFunction passes its negate through; NullFalseLeafBinaryFunction passes negate=false. The bytes each returns are unchanged.
Anything else?
A property test that asserts the helper equals a row-by-row reference through the very accessors it replaces, over every (offset, length) slice of a bitmap whose value and null periods are not multiples of eight and both negate values, pins the offset, validity and negate handling against the definitions it optimizes. No change to any header under include/paimon/, the storage format, or the protocol: ArrowUtils is an internal utility under src/paimon/common/utils/arrow/.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- 主要语言
- C++
- 星标
- 65
- 派生
- 29
- 平均合并
- 2 天 4 小时
- 30 天内合并 PR
- 78
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/paimon-cpp 的其他 Issue
-
enhancement
apache/paimon-cpp#381 · 已指派 1 人 ·
-
难度 4/5 3-5 天 新手友好度 30/100
apache/paimon-cpp#375 · 已指派 1 人 ·
-
enhancement
难度 5/5 一周以上 新手友好度 45/100
apache/paimon-cpp#361 · 已指派 1 人 ·
-
enhancement
难度 4/5 3-5 天 新手友好度 45/100
apache/paimon-cpp#325 · 已指派 1 人 ·
-
enhancement
难度 5/5 一周以上 新手友好度 35/100
apache/paimon-cpp#319 · 1 个 reaction · 已指派 1 人 ·
查看 apache/paimon-cpp 的全部 Issue
相似的 Issue
-
AuTest Bug Tests
难度 2/5 1-3 小时 新手友好度 78/100
apache/trafficserver#13714 ·
-
bug build
难度 2/5 1-3 小时 新手友好度 88/100
facebookincubator/velox#19143 ·
-
难度 2/5 1-3 小时 新手友好度 82/100
-
难度 2/5 1-3 小时 新手友好度 68/100
tenstorrent/tt-metal#57393 · 1 条评论 ·
-
难度 2/5 1-3 小时 新手友好度 76/100
objectionary/eo-graphs#74 ·