Pluggable Backend Interface with DataFusion for Bounded-Memory Compute

実装ファイルやテストは指定されていません。まず関連する issue #1210、#3270、#3554 を読み、次に提案されている ReadBackend、WriteBackend、ComputeBackend プロトコルと DataFusion 統合の範囲を定めてください。完了条件は、一覧にある delete、ストリーミング、ソート、スキャン計画の動作が実現され、DataFusion なしでもリグレッションがなく、PyArrow と DataFusion の結果が同等であることです。

Summary

PyIceberg uses PyArrow as its sole execution engine. PyArrow is a kernel library with no memory management, no spill-to-disk, and no join operators. Operations that process more data than available memory (CoW deletes, equality delete resolution, scan planning for heavily-deleted tables, sorted writes) crash with OOM errors.

This issue tracks introducing a pluggable backend interface (ReadBackend, WriteBackend, ComputeBackend protocols) and integrating Apache DataFusion as the first bounded-memory compute backend.

Problem

Tables written by Flink (which uses equality deletes) are completely unreadable by PyIceberg today.

Solution

1. Pluggable interface: ReadBackend, WriteBackend, ComputeBackend protocols that decouple PyIceberg from PyArrow
2. DataFusion integration: Bounded-memory sort, join, and filter with spill-to-disk via datafusion-python
3. Migration: All existing data operations route through the interface with zero API changes

Deliverables

• Equality delete resolution (NEW): tables with equality deletes can now be read
• CoW delete/overwrite streaming (FIX): statistics short-circuit + two-pass streaming
• Positional delete resolution (IMPROVED): bounded-memory for large delete sets
• Sort-on-write (NEW): external merge sort when DataFusion installed
• Bounded-memory scan planning (NEW): for tables with >100K delete files

Related Issues

• #1210 - Support reading equality delete files
• #3270 - Equality Delete support
• #3554 - Integrate DataFusion as execution engine

Acceptance Criteria

• All existing tests pass without datafusion installed (no regression)
• Tables with equality deletes return correct results
• CoW delete on 2GB+ files completes without OOM (with DataFusion)
• Sort-on-write produces sorted files when table has sort order and DataFusion installed
• Property-based tests verify PyArrow and DataFusion backends produce identical output