hyper: "request exceeding maximum allowed size for request" when reading a Parquet file with large row groups

Offen
  • Stimmt etwas nicht?
#3,294 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
12/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
shell, sql
Bereich
data

Rechercherichtung

The failure comes from Hyper's loader, which is closed source, so the first step is the reproduction in the issue: rewrite hits.parquet with 4M-row row groups using the clickhouse local command shown, then run the hyper/load script on it. The Hyper side cannot be changed from this repository, so a useful contribution is a verified reproduction or a documented limit, not a code patch. Done means the failure is confirmed or ruled out and the row-group limits are recorded.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Hyper fails to load a Parquet file with large row groups:

tableauhyperapi.hyperexception.HyperException: request exceeding maximum allowed size for request:
	line 109, column 6: FROM hits_parquet
	                         ^~~~~~~~~~~~
Context: 0xfa6b0e2f

This happened in the randomized benchmark (https://github.com/ClickHouse/ClickBench/pull/3271), which loads a generated dataset of a billion rows with the schema of hits.parquet, in one file, using hyper/load unchanged (tableauhyperapi 0.0.26963, c6a.metal). The file was written by ClickHouse 26.10 with about 250 row groups of 4M rows each (Snappy compression; the original hits.parquet has 226 row groups of 442K rows). Its footer was about 3.4 MB, so this is a different limit from the size of the footer (https://github.com/ClickHouse/ClickBench/issues/3291). The other systems of the benchmark loaded the same file.

Probable reproduction with the standard dataset (I could not run Hyper here, so it is not verified): rewrite hits.parquet with row groups of 4M rows and load it with hyper/load:

clickhouse local --output_format_parquet_row_group_size 4000000 --output_format_parquet_row_group_size_bytes 100000000000 \
    --query "SELECT * FROM file('hits.parquet') INTO OUTFILE 'hits-large-groups.parquet' TRUNCATE FORMAT Parquet"

Together with the limit of the footer, this bounds how many rows one Parquet file can have for Hyper: too many row groups exceed the footer, too large ones exceed this limit. The randomized benchmark now writes at most a thousand row groups of at least 500K rows (1M rows each for a billion rows) to fit both. What is the limit of the size of a request, and could it be raised or made configurable for reading Parquet?

@caetanosauer

🤖 Generated with Claude Code

Vorherrschende Sprache
Shell
Sterne
1.2k
Forks
334
Ø Merge
3 Std. 50 Min.
Gemergte PRs (30 T.)
710

Entwicklungsumgebung

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus ClickHouse/ClickBench

Alle Issues in ClickHouse/ClickBench

Ähnliche Issues

Weitere Issues zu Shell/Bash

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.