Iceberg: pin tables when a query runs, not when it's looked up

Open
#2,881 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust

Research direction

Start with SessionStateExt::upgrade_for_ballista in ballista/core/src/extension.rs and trace how BallistaQueryPlanner builds DistributedQueryExec; also inspect the scheduler's Flight SQL planning path. Work out a core hook that lets the Iceberg integration pin each distinct catalog table once per query run, then verify reused DataFrames and prepared statements see the latest version and repeated scans share one version.

Written by the indexing model from the issue text.

Description

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

ballista-iceberg (#2638) pins each catalog-backed Iceberg table to one version when a query looks it up, in PinningSchema::table, so the plan can be encoded without I/O. #2784 extends that to views by re-pinning a view's stored plan on each lookup. Choosing the version at lookup still leaves gaps that single-process DataFusion doesn't have, since there IcebergCatalogTableProvider loads its table in scan():

  • A reused DataFrame, or a Flight SQL prepared statement run more than once, reads the version from when it was planned. Neither looks its tables up again.
  • A query that reads a table both through a view and directly looks it up twice, so a commit landing between the two lookups gives them different versions.
  • Lookups do catalog I/O, so anything that only needs a table's name, type or schema (DROP, DESCRIBE, information_schema.columns) loads it, and fails if it can't be loaded. #2784 works around this for views with RepinnedView, and the same problem for base tables is #2882.

Describe the solution you'd like

Pin when a query runs: in the async QueryPlanner::create_physical_plan, walk the whole plan once, load each distinct catalog-backed table once, and swap in a pinned version of each scan. Lookups would then do no I/O, every scan of a table in a query would read one version, and each run of a reused DataFrame or prepared statement would read the latest version. The re-pinning in PinningSchema::table and RepinnedView would go away.

This needs a hook in Ballista's core first. SessionStateExt::upgrade_for_ballista (ballista/core/src/extension.rs) only lets ballista_query_planner() replace BallistaQueryPlanner, so an integration can't wrap it. Either a composable query planner, or a hook that runs on the logical plan before DistributedQueryExec is built, would do. The scheduler plans Flight SQL queries itself, so it needs the same hook.

Describe alternatives you've considered

  • Pin at lookup and re-pin views (#2638, #2784): what exists now, with the gaps above.
  • Re-plan a view from its SQL: SchemaProvider::table gets no session state to plan with, and a view created from a DataFrame has no SQL (#2726).

Additional context

Follow-up to #2638 and #2784.

Dominant language
Rust
Stars
2.2k
Forks
348
Avg merge
2d 4h
Merged PRs (30d)
166

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-ballista

All issues in apache/datafusion-ballista

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.