Vectored Parquet reads can fall back unsafely after partial asynchronous reads and exceed allocation limits
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 45/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 冷清
- 技术栈
- java
调研方向
从 parquet-hadoop 的 ParquetFileReader.java 中第 1293-1307 行附近的向量化读取路径开始,然后跟踪 ChunkListBuilder、异步的兄弟读取以及 parquet.read.allocation.size 的处理。为部分提交或完成、待处理的兄弟 futures、经过过滤的页面、超大列以及启用 checksum 的读取添加回归测试覆盖;完成的标准是能够安全失败或回退,并限制内存分配,同时不改变解码结果。
由索引模型根据 Issue 内容生成。
描述
Describe the bug, including details regarding any error messages, version, and platform.
ParquetFileReader can produce unsafe fallback behavior when Hadoop vectored I/O
is enabled and a filesystem partially submits or completes a vectored read before
raising IllegalArgumentException or UnsupportedOperationException.
The current implementation catches those exceptions around readVectored(...)
and retries every range using ordinary reads against the same ChunkListBuilder.
If an earlier range already populated the builder, its data is appended again.
For a filtered column with selected pages P0 and P2, the buffered page
sequence can become [P0, P0, P2] although the page index still describes
[P0, P2]. Depending on the page contents, decoding can fail or silently
associate the wrong page with the selected rows. Even when no page has been
consumed yet, scalar fallback is unsafe once sibling asynchronous reads may still
be operating on the same stream.
Current upstream code:
The same vectored path also allocates one buffer for an entire contiguous
requested range instead of honoring parquet.read.allocation.size (8 MiB by
default), unlike the ordinary read path. Large projected column chunks or
filtered pages can therefore create unexpectedly large heap allocations.
The current master branch and Apache Parquet Java 1.18.0 contain this behavior.
The vulnerable path is also present in the 1.15.x, 1.16.x, and 1.17.x release
lines. Vectored I/O defaults to enabled starting in 1.16.0, so supported Hadoop
filesystems can reach this path without an explicit opt-in; in 1.15.2 it is
reachable when explicitly enabled.
Expected behavior:
- Preserve ordinary fallback only when vectored I/O is unavailable or range
preparation fails before asynchronous submission starts. - Once submission begins, fail the read safely rather than replaying scalar reads
against a partially populated builder or an active stream. - Wait for already-published sibling reads before returning the original failure.
- Split filesystem byte ranges to respect the configured allocation limit without
changing the logical read plan or decoded results. - Add regression coverage for partial submission/completion, pending sibling
futures, filtered pages, oversized columns, and checksum-enabled reads.
Component(s)
parquet-hadoop
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 6 天 16 小时
- 30 天内合并 PR
- 36
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/parquet-java 的其他 Issue
-
Type: bug
难度 2/5 1-3 小时 新手友好度 68/100
apache/parquet-java#3792 ·
-
难度 2/5 1-3 小时 新手友好度 82/100
apache/parquet-java#3767 ·
-
难度 2/5 1-3 小时 新手友好度 72/100
apache/parquet-java#3695 · 1 条评论 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
apache/parquet-java#3667 ·
-
Type: bug
难度 2/5 1-3 小时 新手友好度 76/100
apache/parquet-java#3587 ·
查看 apache/parquet-java 的全部 Issue
相似的 Issue
-
documentation
难度 2/5 1-3 小时 新手友好度 65/100
inu-appcenter/memorIN-backend#288 ·
-
难度 2/5 1-3 小时 新手友好度 65/100
-
frontend maui-pilot pilot-ask question
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 75/100
-
area/plugin
难度 2/5 1-3 小时 新手友好度 75/100
kestra-io/plugin-kestra#190 ·