Support optional Parquet page index writes
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start by reviewing PR #3829, then inspect _get_parquet_writer_kwargs, mkdocs/docs/configuration.md, and the referenced unit and integration tests. Done means the opt-in property remains false by default, enables PyArrow page-index writing when true, and the documented tests pass.
Written by the indexing model from the issue text.
Description
Feature Request / Improvement
PyIceberg doesn't expose Parquet page index writing, even though PyArrow supports it with ParquetWriter(write_page_index=True).
Use case / motivation
Page indexes let readers such as ClickHouse skip non-matching pages during predicate evaluation, avoiding decoding work for selective queries. Tables written by PyIceberg currently lack these indexes, so page-level pruning is not possible regardless of reader support.
Proposed change
Add an opt-in write.parquet.page-index-enabled table property, default false, to preserve current output and file sizes. When enabled, PyIceberg passes write_page_index=True to PyArrow's ParquetWriter.
Implementation
PR #3829 adds the table property, threads it through _get_parquet_writer_kwargs, documents it in mkdocs/docs/configuration.md, and includes unit and integration test coverage.
Tooling note
I developed this with assistance from DS v4 Pro and reviewed the changes myself.
References
- #3829
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 589
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 72
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3996 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#3979 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg-python#3866 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3836 · 1 comment ·
All issues in apache/iceberg-python
Similar issues
-
bug ci good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
documentation
Difficulty 2/5 Half a day Newbie friendliness 62/100
inmanta/inmanta-core#10835 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
sponsored
Difficulty 2/5 1-3 hours Newbie friendliness 65/100