Enable stats collection for nested fields and use write.metadata.metrics.max-inferred-column-defaults to control stats growth
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 66/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start in pyiceberg/io/pyarrow.py around the linked lines that disable statistics for nested fields, then trace how write.metadata.metrics.max-inferred-column-defaults is handled. Done means nested struct fields can collect lower and upper bounds while the table property controls inferred-column statistics growth; use the existing statistics-related tests, if found nearby, to verify both behaviors.
Written by the indexing model from the issue text.
Description
Feature Request / Improvement
I recently discovered that full stats collection (i.e. lower_bounds/upper_bounds) is explicitly disabled in PyIceberg for nested (i.e. struct child) fields.
This change was made in this PR and specifically this commit.
It seems that this change may have been made to limit the number of fields whose stats are collected when default-full stats collection is enabled. However, after discussion it seems that simply adding support for the write.metadata.metrics.max-inferred-column-defaults table property would be the preferred way to control stats growth. If this is implemented, re-enabling stats collection for nested fields should be a non-issue.
Stats collection for nested struct fields is important for schemas like GeoParquet which store important primitive fields (in this case, bounding box xmin, ymin, xmax, ymax) using structs.
See also this slack thread for discussion.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 589
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 70
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
kind:bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
apache/iceberg-python#4006 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3996 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#3979 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg-python#3866 · 1 comment ·
All issues in apache/iceberg-python
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
StevenBlack/hosts#3256 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
qualcomm/qai-appbuilder#275 ·