Parquet import of wide-string files (websight-v01) runs out of heap at -Xmx8g

Aberta
  • Algo errado?
#657 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
35/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
cli, data-engineering

Direção de pesquisa

Start from ParquetImporter and ImportOptions#withChunkSize to see how rows are batched before they reach hardwood, then read the stack trace's PageDecoder.parseDataPage and ColumnWorker.decode in hardwood. Reproducing needs the 29 GB websight-v01 file, so first measure the heap at a smaller chunk size. Done means a bounded-heap import succeeds at a stated -Xmx, with the chunk-size or CLI option documented.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug

vortex import of Raincloud's websight-v01 (29 GB Parquet, 822,987 rows, ~35 KB of HTML per row) runs out of heap with -Xmx8g, 15% of the way through:

java -Xmx8g --enable-native-access=ALL-UNNAMED -jar vortex-cli-0.19.1-SNAPSHOT-all.jar import websight-v01.parquet out.vortex
  [====>   ]  15%  131,072 / 822,987 rows
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOfRange(Arrays.java:3849)
	at dev.hardwood.internal.encoding.PlainDecoder.readByteArray(PlainDecoder.java:285)
	at dev.hardwood.internal.encoding.PlainDecoder.readByteArrays(PlainDecoder.java:217)
	at dev.hardwood.internal.reader.PageDecoder.decodeTypedValues(PageDecoder.java:485)
	at dev.hardwood.internal.reader.PageDecoder.parseDataPage(PageDecoder.java:335)
	at dev.hardwood.internal.reader.ColumnWorker.decode(ColumnWorker.java:508)

The same import with -Xmx48g succeeds (822,987 rows, 77 s, 30.9 GB output), so this is a memory-footprint problem, not a correctness one. The failure is in hardwood's page decode (one byte[] per value, held for the whole page), reached through ParquetImporter.

Unknown, to measure:

  • how much of the heap is hardwood's decoded pages vs the importer's own chunk buffers (default ImportOptions chunk size is 65,536 rows, which is ~2.3 GB of strings for this file before encoding);
  • the smallest -Xmx that works, and whether a smaller chunk size (ImportOptions#withChunkSize) lowers it. The CLI has no flag for the chunk size today.
  • whether hardwood decodes columns in parallel and so multiplies the footprint (the file has three columns: image struct of binary + path, text, split).

Hits any wide-string Parquet: found while rerunning the #643 slugs (websight-v01, open-food-facts; the latter imported fine with -Xmx48g). Not an -Xmx default problem on a large machine (the default heap is a quarter of RAM), but a laptop with 16 GB would hit it.

Possible directions: size the chunk by bytes as well as rows, so a 35 KB-per-row file uses fewer rows per chunk; expose the chunk size on the CLI; document the heap a large import needs.

Linguagem predominante
Java
Estrelas
3
Forks
2
Merge médio
9h 51min
PRs com merge (30d)
127

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de dfa1/vortex-java

Todas as issues de dfa1/vortex-java

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.