hyper: "request exceeding maximum allowed size for request" when reading a Parquet file with large row groups

Aberta
  • Algo errado?
#3,294 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
12/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
shell, sql
Domínio
data

Direção de pesquisa

The failure comes from Hyper's loader, which is closed source, so the first step is the reproduction in the issue: rewrite hits.parquet with 4M-row row groups using the clickhouse local command shown, then run the hyper/load script on it. The Hyper side cannot be changed from this repository, so a useful contribution is a verified reproduction or a documented limit, not a code patch. Done means the failure is confirmed or ruled out and the row-group limits are recorded.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Hyper fails to load a Parquet file with large row groups:

tableauhyperapi.hyperexception.HyperException: request exceeding maximum allowed size for request:
	line 109, column 6: FROM hits_parquet
	                         ^~~~~~~~~~~~
Context: 0xfa6b0e2f

This happened in the randomized benchmark (https://github.com/ClickHouse/ClickBench/pull/3271), which loads a generated dataset of a billion rows with the schema of hits.parquet, in one file, using hyper/load unchanged (tableauhyperapi 0.0.26963, c6a.metal). The file was written by ClickHouse 26.10 with about 250 row groups of 4M rows each (Snappy compression; the original hits.parquet has 226 row groups of 442K rows). Its footer was about 3.4 MB, so this is a different limit from the size of the footer (https://github.com/ClickHouse/ClickBench/issues/3291). The other systems of the benchmark loaded the same file.

Probable reproduction with the standard dataset (I could not run Hyper here, so it is not verified): rewrite hits.parquet with row groups of 4M rows and load it with hyper/load:

clickhouse local --output_format_parquet_row_group_size 4000000 --output_format_parquet_row_group_size_bytes 100000000000 \
    --query "SELECT * FROM file('hits.parquet') INTO OUTFILE 'hits-large-groups.parquet' TRUNCATE FORMAT Parquet"

Together with the limit of the footer, this bounds how many rows one Parquet file can have for Hyper: too many row groups exceed the footer, too large ones exceed this limit. The randomized benchmark now writes at most a thousand row groups of at least 500K rows (1M rows each for a billion rows) to fit both. What is the limit of the size of a request, and could it be raised or made configurable for reading Parquet?

@caetanosauer

🤖 Generated with Claude Code

Linguagem predominante
Shell
Estrelas
1.2k
Forks
334
Merge médio
3h 50min
PRs com merge (30d)
710

Preparar o ambiente

  • Sem Dockerfile nem arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de ClickHouse/ClickBench

Todas as issues de ClickHouse/ClickBench

Issues semelhantes

Mais issues de Shell/Bash

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.