Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Refactor
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- performance
Research direction
Start in pyiceberg/avro/file.py at AvroFileHeader.get_schema() and trace the avro_to_iceberg conversion. Use scan().plan_files() with repeated manifests to observe the repeated work and confirm that conversion results are reused by schema string, with planning performance improved as manifest count grows.
Written by the indexing model from the issue text.
Description
Feature Request / Improvement
AvroFileHeader.get_schema() (pyiceberg/avro/file.py) runs the full avro_to_iceberg conversion every time an
Avro file is opened.
The inefficiency
- Every manifest under a spec embeds an identical Avro schema string.
- So during scan planning, that same conversion is repeated once per manifest.
- The cost grows with the manifest count, even though only a couple of distinct schemas are ever involved.
Proposed fix
- The conversion depends only on the schema string, so the result can be cached (keyed on that string).
Measured impact
scan().plan_files()on an unpartitioned 150-manifest table: ~86 ms → ~48 ms (~1.8× faster).- The saving grows as the number of manifests increases.
I am willing to contribute for this improvement.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 589
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 72
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3996 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#3979 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg-python#3866 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#3836 · 1 comment ·
All issues in apache/iceberg-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100