hyper: "request exceeding maximum allowed size for request" when reading a Parquet file with large row groups
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
The failure comes from Hyper's loader, which is closed source, so the first step is the reproduction in the issue: rewrite hits.parquet with 4M-row row groups using the clickhouse local command shown, then run the hyper/load script on it. The Hyper side cannot be changed from this repository, so a useful contribution is a verified reproduction or a documented limit, not a code patch. Done means the failure is confirmed or ruled out and the row-group limits are recorded.
由索引模型根据 Issue 内容生成。
描述
Hyper fails to load a Parquet file with large row groups:
tableauhyperapi.hyperexception.HyperException: request exceeding maximum allowed size for request:
line 109, column 6: FROM hits_parquet
^~~~~~~~~~~~
Context: 0xfa6b0e2f
This happened in the randomized benchmark (https://github.com/ClickHouse/ClickBench/pull/3271), which loads a generated dataset of a billion rows with the schema of hits.parquet, in one file, using hyper/load unchanged (tableauhyperapi 0.0.26963, c6a.metal). The file was written by ClickHouse 26.10 with about 250 row groups of 4M rows each (Snappy compression; the original hits.parquet has 226 row groups of 442K rows). Its footer was about 3.4 MB, so this is a different limit from the size of the footer (https://github.com/ClickHouse/ClickBench/issues/3291). The other systems of the benchmark loaded the same file.
Probable reproduction with the standard dataset (I could not run Hyper here, so it is not verified): rewrite hits.parquet with row groups of 4M rows and load it with hyper/load:
clickhouse local --output_format_parquet_row_group_size 4000000 --output_format_parquet_row_group_size_bytes 100000000000 \
--query "SELECT * FROM file('hits.parquet') INTO OUTFILE 'hits-large-groups.parquet' TRUNCATE FORMAT Parquet"
Together with the limit of the footer, this bounds how many rows one Parquet file can have for Hyper: too many row groups exceed the footer, too large ones exceed this limit. The randomized benchmark now writes at most a thousand row groups of at least 500K rows (1M rows each for a billion rows) to fit both. What is the limit of the size of a request, and could it be raised or made configurable for reading Parquet?
@caetanosauer
🤖 Generated with Claude Code
- 主要语言
- Shell
- 星标
- 1.2k
- 派生
- 334
- 平均合并
- 3 小时 50 分钟
- 30 天内合并 PR
- 710
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
ClickHouse/ClickBench 的其他 Issue
-
难度 4/5 3-5 天 新手友好度 25/100
ClickHouse/ClickBench#3292 ·
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 15/100
ClickHouse/ClickBench#3291 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 5/100
ClickHouse/ClickBench#3288 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 35/100
ClickHouse/ClickBench#2650 ·
维护者通常 1 天内回复
-
doris-parquet-single fails before measurement on c6a.xlarge: FE cannot commit its 8 GiB initial heap未关闭
难度 4/5 3-5 天 新手友好度 35/100
ClickHouse/ClickBench#2208 · 2 条评论 ·
维护者通常 1 天内回复
查看 ClickHouse/ClickBench 的全部 Issue
相似的 Issue
-
good first issue help wanted
难度 2/5 1-3 小时 新手友好度 82/100
-
documentation good first issue
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复
-
bug good first issue help wanted
难度 2/5 1-3 小时 新手友好度 82/100
laaielichter/Godotwright#213 ·
维护者通常 1 天内回复
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 84/100
laaielichter/AgentHaven#72 ·
维护者通常 1 天内回复
-
type: cleanup
难度 2/5 1-3 小时 新手友好度 85/100
googleapis/google-cloud-swift#1626 ·
维护者通常 1 天内回复