[EPIC] Faster manifest reads through pyiceberg-core

Start with the task list and the referenced entry points: ManifestWriter.add_entry in pyiceberg/manifest.py, inspect.entries() and inspect.files() in pyiceberg/table/inspect.py, and _open_manifest in pyiceberg/table/init.py. Read iceberg-rust#3262 and confirm the required pyiceberg-core Arrow output and release dependency. Done means the selected manifest-reading paths use the faster output while retaining the Cython fallback and matching existing results and benchmarks.

Feature Request / Improvement

PyIceberg reads manifests and manifest lists with its Cython Avro decoder. pyiceberg-core (the iceberg-rust Python binding) already exposes read_manifest_entries and read_manifest_list, but using them today makes manifest reads 4x to 5x slower than Cython. This issue tracks the work in both repos to make the pyiceberg-core path faster than Cython, and then to switch PyIceberg to it. The iceberg-rust side is tracked in apache/iceberg-rust#3262, which repeats those tasks in more detail.

The earlier attempt (apache/iceberg-rust#1280, apache/iceberg-python#2493) was closed. It converted every Rust manifest entry into a PyIceberg DataFile in Python, which is the rust_convert path measured below. The full benchmark writeup from @kevinjqliu, including an in-crate profile of the Rust parser and the script, is in https://github.com/kevinjqliu/iceberg-python/issues/45.

Where the time goes

These numbers come from manifests written by PyIceberg (V2, deflate, 12 columns with full column stats, identity partition). pyiceberg is at main (0d58407) with the Cython decoder. pyiceberg-core is built locally from the v0.10.1 tag, once as released (opt-level = "z") and once with opt-level = 3. Each number is the best of 7 runs in ms, single threaded, on Apple Silicon with Python 3.12. v0.10.1 predates apache/iceberg-rust#3028, which its PR measured as making the Rust parse about 5% faster.

The pyiceberg-core columns include building PyIceberg ManifestEntry and DataFile objects, so all three columns produce the same result. At 10,000 entries with opt-level = 3, the Rust parse costs 21.7 us per entry and converting the result into PyIceberg objects costs another 10.3 us per entry. The whole Cython path costs 10.3 us per entry in the same run. The conversion alone costs as much as Cython, so a faster Rust parser can't reach parity unless PyIceberg also stops building one Python object per entry from the binding's output.

Manifest lists are closer. With opt-level = 3 the Rust parse of a 10,000-entry manifest list takes 8.0 ms against 23.9 ms for Cython, and the conversion into ManifestFile objects is what makes the end-to-end path slower (39.1 ms).

Task list

These tasks are in PyIceberg.

• Write equality_ids as list<int> per the spec. PR: #3842.
• Flush manifest entries in size-bounded Avro blocks. ManifestWriter.add_entry calls write_block once per entry (manifest.py#L1213-L1238), so every entry gets its own deflate stream and sync marker. Java writes manifests through Avro's DataFileWriter and keeps its default sync interval, which the linked issue reports as 64 KB. In the linked benchmark, a 10,000-entry manifest shrank from 3.6 MB to 0.7 MB and read about 1.5x faster with both decoders after re-encoding with 64 KB blocks. The manifest-list writer already writes one block (manifest.py#L1389). This task doesn't depend on anything else. PR: https://github.com/apache/iceberg-python/pull/4008
• Build inspect.entries() and inspect.files() from pyiceberg-core Arrow output. Both methods build a DataFile per entry and then flatten them back into a pa.Table (inspect.py#L152, inspect.py#L860). These are the first consumer of the Arrow output from the iceberg-rust tasks, because they want Arrow anyway. This task depends on that output being in a pyiceberg-core release.
• Plan scans through pyiceberg-core. _open_manifest (table/__init__.py#L2321-L2334) filters each entry with Python partition and metrics evaluators while decoding (#3658), but every entry still becomes a DataFile before it is filtered. To beat Cython, filtering has to happen on Arrow columns or inside Rust, so that only the matching entries become Python objects. Which of the two to use is open and is settled together with the iceberg-rust task for filtering. pyiceberg-core is an optional extra, so the Cython reader stays as the fallback.
• Read manifest lists through pyiceberg-core. This uses the same approach as scan planning, applied to read_manifest_list.

These tasks are in iceberg-rust and tracked in apache/iceberg-rust#3262.

• Faster manifest parsing. Upgrade to apache-avro 0.22 and decode entries directly from the writer schema instead of through apache_avro::Value and schema resolution (apache/iceberg-rust#3063).
• Build the pyiceberg-core wheel with opt-level = 3. In the numbers above, this makes the parse about 1.6x faster and the end-to-end path about 1.4x faster. It also nearly doubles the wheel size.
• Return manifest entries and manifest lists as Arrow from pyiceberg-core.
• Release pyiceberg-core with the above. PyIceberg pins pyiceberg-core>=0.10.1,<0.11.0.