umbra: the server crashes while loading a billion-row Parquet file with a large footer

Ouverte
  • Quelque chose ne va pas ?
#3,292 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
25/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
shell, sql
Domaine
data, database

Piste de recherche

The crash happens during the COPY in umbra/create.sql (around line 216) when umbra/load reads a Parquet file with about 2,360 row groups and a 31.6 MB footer. Start by rebuilding the reduced-row-group file with the clickhouse local command in the issue, then load it with umbra/load and capture the Umbra server log, which the report did not include. Done means the crash reproduces or is ruled out, with the footer size and row-group count recorded.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

The Umbra server crashed while loading a Parquet file of a billion rows: the client lost the connection during the COPY of umbra/create.sql:

CREATE TABLE
psql:create.sql:216: SSL SYSCALL error: EOF detected
psql:create.sql:216: error: connection to server was lost

This happened in the randomized benchmark (https://github.com/ClickHouse/ClickBench/pull/3271), which loads a generated dataset with the same schema as hits.parquet, a billion rows in one file, using umbra/install and umbra/load unchanged (image umbradb/umbra:26.10, machine c6a.metal with 384 GiB of memory). The file was written by ClickHouse 26.10 with its default row group size limit of 512 MB: more than a thousand row groups, and a footer of 35 MB. Hyper refused the same file because of its footer size (https://github.com/ClickHouse/ClickBench/issues/3291), so the size of the footer is a likely cause here too; the other systems of the benchmark loaded the same file. The log of the server was not collected.

Possible reproduction with the standard dataset: rewriting hits.parquet (100M rows, 226 row groups, a footer of 2.4 MB) with smaller row groups gives 2360 row groups and a footer of 31.6 MB:

clickhouse local --output_format_parquet_row_group_size 40000 --output_format_parquet_row_group_size_bytes 100000000000 \
    --query "SELECT * FROM file('hits.parquet') INTO OUTFILE 'hits-small-groups.parquet' TRUNCATE FORMAT Parquet"
clickhouse local --query "SELECT num_row_groups, metadata_size FROM file('hits-small-groups.parquet', ParquetMetadata)"
# 2360	31626081

then load it instead of hits.parquet with umbra/load.

The next run of the randomized benchmark writes about 250 row groups per file, as the original file has; its result will show whether Umbra loads a billion rows with a small footer, and I will add it here.

@toschmidt

🤖 Generated with Claude Code

Langage dominant
Shell
Étoiles
1.2k
Forks
334
Merge moyen
3 h 50 min
PR mergées (30 j)
710

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de ClickHouse/ClickBench

Toutes les issues de ClickHouse/ClickBench

Issues similaires

Plus d'issues Shell/Bash

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.