Parquet export fails on struct and list columns

Open
  • Is something wrong?
#721 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java
Domain
data-engineering

Research direction

Start in ParquetExporter.addColumn (ParquetExporter.java around line 179-180), where the STRUCT/LIST/MAP check throws, and trace how exportParquet (line 125) builds the schema for each column; the CLI path is ExportCommand.runParquet. Map the List, struct and Map types from the issue to Parquet group types, with Variant needing its own decision. Done means the four Raincloud datasets round-trip Parquet to Vortex to Parquet, or, as a smaller first step, the error becomes a VortexException naming the column.

Written by the indexing model from the issue text.

Description

bug

ParquetExporter (and so vortex export x.vortex out.parquet) refuses any Struct, List/FixedSizeList or Map column:

java.lang.UnsupportedOperationException: unsupported column type for Parquet export (STRUCT/LIST/MAP not yet supported): List[elementType=Primitive[ptype=F32, nullable=true], nullable=true] (column: emb)
	at io.github.dfa1.vortex.parquet.ParquetExporter.addColumn(ParquetExporter.java:179)
	at io.github.dfa1.vortex.parquet.ParquetExporter.exportParquet(ParquetExporter.java:125)
	at io.github.dfa1.vortex.cli.ExportCommand.runParquet(ExportCommand.java:102)

Found with the released 0.19.1 CLI, exporting Vortex files imported from Raincloud Parquet: 4 of 10 public datasets fail, and the code on main is the same (ParquetExporter.java:180).

Raincloud slug Column Type
cohere-wikipedia-simple-embed emb fixed_size_list<float>[1024] (read as List<F32>)
coig conversations list<struct<question: string, answer: string>>
countries-of-the-world data struct<metadata: binary, value: binary> (Parquet Variant)
mnist image struct<bytes: binary, path: string>

Repro: vortex import <slug>.parquet x.vortex && vortex export x.vortex x.parquet. The import of the same Parquet works, so a Parquet → Vortex → Parquet round trip is lossy only for these types.

Expected: nested columns export as Parquet groups (LIST / plain group / MAP logical types), so that round trip holds. Until then the failure should be a VortexException naming the column, not an UnsupportedOperationException.

Dominant language
Java
Stars
3
Forks
2
Avg merge
9h 19m
Merged PRs (30d)
154

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dfa1/vortex-java

All issues in dfa1/vortex-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.