fix(streaming-write): use rolling ParquetWriter + OutputStream.tell() for spec-correct file sizes and bounded memory
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start at Table.append/Table.overwrite for RecordBatchReader input and trace the bin_pack_record_batches path, then read the proposed pq.ParquetWriter and OutputStream.tell() usage. Done means target_file_size_bytes reflects compressed on-disk size and memory stays bounded without changing the public API; the issue names no test file.
Written by the indexing model from the issue text.
Description
Background
PR #3335 added pa.RecordBatchReader as a valid input to Table.append/Table.overwrite using a buffered bin-pack approach (bin_pack_record_batches). That implementation has two acknowledged caveats called out in its docstrings:
- Memory bound: peak memory is
N_workers × write.target-file-size-bytes(~4 GiB at defaults) — better than materialising everything, but not constant. - Byte semantics:
write.target-file-size-bytesis interpreted as uncompressed in-memory Arrow bytes, not on-disk compressed Parquet bytes. Resulting files are typically 3–10× smaller than the property suggests — diverging from the Java/Spark/Flink writers.
Proposed fix
Replace the bin-pack approach with a rolling pq.ParquetWriter driven by OutputStream.tell() (added in #2998 specifically for this purpose):
with output_file.create(overwrite=True) as fos:
with pq.ParquetWriter(fos, schema=..., ...) as writer:
writer.write_batch(first_batch)
while fos.tell() < target_file_size: # ← compressed on-disk bytes
batch = next(batches)
writer.write_batch(batch)
This delivers:
- Spec-correct file sizes:
tell()reports compressed on-disk bytes, sowrite.target-file-size-bytesfinally means what the Iceberg spec intends — consistent with the Java/Spark/Flink writers. - Truly bounded memory: peak RSS is bounded by one input batch + Parquet page buffer (~1 MiB × columns) + S3 multipart pool (~5 MiB × ~8 parts), regardless of
target_file_size, dataset size, or number of files produced. - No public API change: same
tbl.append(reader)/tbl.overwrite(reader)interface.
Fix
#3336
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 589
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 70
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
kind:bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
apache/iceberg-python#4006 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3996 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#3979 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg-python#3866 · 1 comment ·
All issues in apache/iceberg-python
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
learningequality/ricecooker#747 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
run-llama/llama_index#23199 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
KhronosGroup/glTF-Blender-IO#2769 ·