Optimize parquet_files_to_data_files by reusing schema index

Open Beginner friendly
#2,032 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
performance

Research direction

Start in crates/iceberg/src/writer/file_writer/parquet_writer.rs at parquet_files_to_data_files and parquet_to_data_file_builder, then inspect IndexByParquetPathName creation and its use. The work is done when the schema index is built once before the file loop, passed by reference to the builder, and reused for every file without changing the resulting DataFile values.

Written by the indexing model from the issue text.

Description

In the current implementation of parquet_files_to_data_files in parquet_writer.rs, we iterate over a list of files to convert them into Iceberg
DataFile structs.

However, for every single file in the loop, we invoke
parquet_to_data_file_builder
, which internally rebuilds the schema index from scratch:


// crates/iceberg/src/writer/file_writer/parquet_writer.rs

pub(crate) fn parquet_to_data_file_builder(...) -> Result<DataFileBuilder> {
    // This runs for every file!
    let index_by_parquet_path = {
        let mut visitor = IndexByParquetPathName::new();
        visit_schema(&schema, &mut visitor)?;
        visitor
    };
    
    // ...
}

Problem: When importing a large number of files (e.g., thousands of files in a bulk import), we are traversing the entire schema and allocating a new name_to_id HashMap thousands of times, even though the schema is constant for the entire operation.

Proposed Solution:

Dominant language
Rust
Stars
1.4k
Forks
574
Avg merge
1d 18h
Merged PRs (30d)
84

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-rust

All issues in apache/iceberg-rust

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.