IO: Consolidate PyArrow logic into io/pyarrow.py before decomposition
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
Research direction
Start by comparing table/upsert_util.py with pyiceberg/io/pyarrow.py and reviewing the related discussion in #3737. Focus on PR A first; done means the upsert utility's PyArrow logic routes through io/pyarrow.py, with no behavior changes and all existing tests passing.
Written by the indexing model from the issue text.
Description
Summary
Before decomposing pyiceberg/io/pyarrow.py into focused submodules (#3737, #3738), we should consolidate PyArrow-specific logic that currently lives outside the module. This ensures all PyArrow calls route through a single boundary, making the subsequent split clean and enabling future engine substitution.
Motivation
Per discussion in #3737, @rambleraptor noted that the first useful step is ensuring no PyArrow logic occurs outside pyarrow.py. Currently several modules import pyarrow directly and implement compute logic inline rather than delegating through pyiceberg.io.pyarrow.
When we later introduce a ComputeEngine protocol, any PyArrow logic outside the module boundary bypasses the protocol and prevents clean substitution.
Audit
Grepped pyiceberg/ (excluding io/pyarrow.py and tests) for runtime import pyarrow statements (both top-level and inline). Excluded TYPE_CHECKING-only imports since those have no runtime dependency.
| Location | What it does | Action |
|---|---|---|
table/upsert_util.py |
PyArrow table joins, group_by, compute, cast, take | Absorb |
table/inspect.py |
Builds pa.schema + pa.Table.from_pylist for metadata inspection | TBD |
transforms.py |
pyarrow_transform() dispatch on pa.Array/ChunkedArray |
TBD |
table/__init__.py |
Entry points accept pa.Table, delegate to io.pyarrow | Leave |
table/deletion_vector.py |
Single pa.chunked_array() call | Leave |
catalog/__init__.py |
Delegates to io.pyarrow for schema conversion | Leave |
Plan
One PR per absorption. Each is a pure refactor: move code into io/pyarrow.py, have the caller import from pyiceberg.io.pyarrow instead of pyarrow directly. No behavior change, all existing tests pass unchanged.
- PR A: Absorb
table/upsert_util.pyPyArrow logic - PR B:
table/inspect.py(pending discussion) - PR C:
transforms.py(pending discussion)
Related
- #3737 - Decompose io/pyarrow.py into focused modules
- #3738 - Extract PyArrowFileIO (first decomposition step)
- #3715 / #3716 - Previous pluggable backend attempt (rejected as too large)
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 589
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 72
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3996 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#3979 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg-python#3866 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3836 · 1 comment ·
All issues in apache/iceberg-python
Similar issues
-
Add: hunch Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
AbdelStark/awesome-typesafe#104 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
DiamondLightSource/dodal#2211 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
openml/openml-python#1749 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
sipyourdrink-ltd/bernstein#6191 ·