Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

fix(streaming-write): use rolling ParquetWriter + OutputStream.tell() for spec-correct file sizes and bounded memory

Open
#3,388 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
data

Research direction

Start at Table.append/Table.overwrite for RecordBatchReader input and trace the bin_pack_record_batches path, then read the proposed pq.ParquetWriter and OutputStream.tell() usage. Done means target_file_size_bytes reflects compressed on-disk size and memory stays bounded without changing the public API; the issue names no test file.

Written by the indexing model from the issue text.

Description

Background

PR #3335 added pa.RecordBatchReader as a valid input to Table.append/Table.overwrite using a buffered bin-pack approach (bin_pack_record_batches). That implementation has two acknowledged caveats called out in its docstrings:

  1. Memory bound: peak memory is N_workers × write.target-file-size-bytes (~4 GiB at defaults) — better than materialising everything, but not constant.
  2. Byte semantics: write.target-file-size-bytes is interpreted as uncompressed in-memory Arrow bytes, not on-disk compressed Parquet bytes. Resulting files are typically 3–10× smaller than the property suggests — diverging from the Java/Spark/Flink writers.

Proposed fix

Replace the bin-pack approach with a rolling pq.ParquetWriter driven by OutputStream.tell() (added in #2998 specifically for this purpose):

with output_file.create(overwrite=True) as fos:
    with pq.ParquetWriter(fos, schema=..., ...) as writer:
        writer.write_batch(first_batch)
        while fos.tell() < target_file_size:   # ← compressed on-disk bytes
            batch = next(batches)
            writer.write_batch(batch)

This delivers:

  • Spec-correct file sizes: tell() reports compressed on-disk bytes, so write.target-file-size-bytes finally means what the Iceberg spec intends — consistent with the Java/Spark/Flink writers.
  • Truly bounded memory: peak RSS is bounded by one input batch + Parquet page buffer (~1 MiB × columns) + S3 multipart pool (~5 MiB × ~8 parts), regardless of target_file_size, dataset size, or number of files produced.
  • No public API change: same tbl.append(reader) / tbl.overwrite(reader) interface.

Fix

#3336

Dominant language
Python
Stars
1.1k
Forks
589
Avg merge
2d 2h
Merged PRs (30d)
70

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-python

All issues in apache/iceberg-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.