Iceberg: pin tables when a query runs, not when it's looked up
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
Research direction
Start with SessionStateExt::upgrade_for_ballista in ballista/core/src/extension.rs and trace how BallistaQueryPlanner builds DistributedQueryExec; also inspect the scheduler's Flight SQL planning path. Work out a core hook that lets the Iceberg integration pin each distinct catalog table once per query run, then verify reused DataFrames and prepared statements see the latest version and repeated scans share one version.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
ballista-iceberg (#2638) pins each catalog-backed Iceberg table to one version when a query looks it up, in PinningSchema::table, so the plan can be encoded without I/O. #2784 extends that to views by re-pinning a view's stored plan on each lookup. Choosing the version at lookup still leaves gaps that single-process DataFusion doesn't have, since there IcebergCatalogTableProvider loads its table in scan():
- A reused
DataFrame, or a Flight SQL prepared statement run more than once, reads the version from when it was planned. Neither looks its tables up again. - A query that reads a table both through a view and directly looks it up twice, so a commit landing between the two lookups gives them different versions.
- Lookups do catalog I/O, so anything that only needs a table's name, type or schema (
DROP,DESCRIBE,information_schema.columns) loads it, and fails if it can't be loaded. #2784 works around this for views withRepinnedView, and the same problem for base tables is #2882.
Describe the solution you'd like
Pin when a query runs: in the async QueryPlanner::create_physical_plan, walk the whole plan once, load each distinct catalog-backed table once, and swap in a pinned version of each scan. Lookups would then do no I/O, every scan of a table in a query would read one version, and each run of a reused DataFrame or prepared statement would read the latest version. The re-pinning in PinningSchema::table and RepinnedView would go away.
This needs a hook in Ballista's core first. SessionStateExt::upgrade_for_ballista (ballista/core/src/extension.rs) only lets ballista_query_planner() replace BallistaQueryPlanner, so an integration can't wrap it. Either a composable query planner, or a hook that runs on the logical plan before DistributedQueryExec is built, would do. The scheduler plans Flight SQL queries itself, so it needs the same hook.
Describe alternatives you've considered
- Pin at lookup and re-pin views (#2638, #2784): what exists now, with the gaps above.
- Re-plan a view from its SQL:
SchemaProvider::tablegets no session state to plan with, and a view created from aDataFramehas no SQL (#2726).
Additional context
Follow-up to #2638 and #2784.
- Dominant language
- Rust
- Stars
- 2.2k
- Forks
- 348
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 166
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion-ballista
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-ballista#2885 · 1 reaction ·
Maintainers usually reply within 1 day
-
Executor's default work dir is deleted at startup and recreated without `TempDir`'s permissionsPossibly taken @aetherxeg-source claimed this 16 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-ballista#2767 ·
Maintainers usually reply within 1 day
-
Release a task's vcores as its partitions finishPossibly taken @Dandandan claimed this today. Openenhancement
Difficulty 5/5 Over a week Newbie friendliness 18/100
apache/datafusion-ballista#2905 ·
Maintainers usually reply within 1 day
-
Iceberg: DROP TABLE fails on a registered table its catalog can't loadPossibly taken @NoahKusaba claimed this 15 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 43/100
apache/datafusion-ballista#2882 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
apache/datafusion-ballista#2872 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/datafusion-ballista
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
antithesishq/bombadil#473 ·
Maintainers usually reply within 2 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
libracoder/kiri#669 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 5 days
-
autofixer: follow-up
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
tursodatabase/turso#12217 ·
Maintainers usually reply within 1 day
-
Declare the CMS schema library in the optional recipients extraPossibly taken A pull request linked to this issue is open or already merged. Openarea:python in-progress
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
yhay81/pyropdf#1005 · 2 comments ·
Maintainers usually reply within 1 day