hyper: "request exceeding maximum allowed size for request" when reading a Parquet file with large row groups

Ouverte
  • Quelque chose ne va pas ?
#3,294 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
12/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
shell, sql
Domaine
data

Piste de recherche

The failure comes from Hyper's loader, which is closed source, so the first step is the reproduction in the issue: rewrite hits.parquet with 4M-row row groups using the clickhouse local command shown, then run the hyper/load script on it. The Hyper side cannot be changed from this repository, so a useful contribution is a verified reproduction or a documented limit, not a code patch. Done means the failure is confirmed or ruled out and the row-group limits are recorded.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Hyper fails to load a Parquet file with large row groups:

tableauhyperapi.hyperexception.HyperException: request exceeding maximum allowed size for request:
	line 109, column 6: FROM hits_parquet
	                         ^~~~~~~~~~~~
Context: 0xfa6b0e2f

This happened in the randomized benchmark (https://github.com/ClickHouse/ClickBench/pull/3271), which loads a generated dataset of a billion rows with the schema of hits.parquet, in one file, using hyper/load unchanged (tableauhyperapi 0.0.26963, c6a.metal). The file was written by ClickHouse 26.10 with about 250 row groups of 4M rows each (Snappy compression; the original hits.parquet has 226 row groups of 442K rows). Its footer was about 3.4 MB, so this is a different limit from the size of the footer (https://github.com/ClickHouse/ClickBench/issues/3291). The other systems of the benchmark loaded the same file.

Probable reproduction with the standard dataset (I could not run Hyper here, so it is not verified): rewrite hits.parquet with row groups of 4M rows and load it with hyper/load:

clickhouse local --output_format_parquet_row_group_size 4000000 --output_format_parquet_row_group_size_bytes 100000000000 \
    --query "SELECT * FROM file('hits.parquet') INTO OUTFILE 'hits-large-groups.parquet' TRUNCATE FORMAT Parquet"

Together with the limit of the footer, this bounds how many rows one Parquet file can have for Hyper: too many row groups exceed the footer, too large ones exceed this limit. The randomized benchmark now writes at most a thousand row groups of at least 500K rows (1M rows each for a billion rows) to fit both. What is the limit of the size of a request, and could it be raised or made configurable for reading Parquet?

@caetanosauer

🤖 Generated with Claude Code

Langage dominant
Shell
Étoiles
1.2k
Forks
334
Merge moyen
3 h 50 min
PR mergées (30 j)
710

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de ClickHouse/ClickBench

Toutes les issues de ClickHouse/ClickBench

Issues similaires

Plus d'issues Shell/Bash

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.