Support multiple output partitions for lazy file discovery

Open
  • Is something wrong?
#84 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
data-engineering, distributed-systems

Research direction

Start with the eager planning work in pull request #67 and its linked review discussion, then trace the lazy file discovery and execution paths. Define how partitions share discovery, assign tasks exactly once, handle early cancellation and backpressure, and isolate state across plan executions while preserving shared delete-loading state. Validate complete and filtered results, position deletes, LIMIT cancellation, and repeated executions.

Written by the indexing model from the issue text.

Description

Lazy Iceberg scans currently expose one DataFusion output partition. Eager planning in #67 can distribute retained file tasks across local partitions, but partitioned execution should also be possible without collecting all tasks first.

Support distributing one lazy file-discovery stream across multiple local output partitions. The design should avoid independently listing the same manifests for every partition, preserve exactly-once task assignment within each execution, and define cancellation/backpressure behavior when partitions stop early or are never opened. Re-executing a physical plan must get independent execution state. Reader clones should continue sharing delete-loading state within an execution.

Validate complete and filtered results, position deletes, LIMIT/cancellation, and repeated executions. This is a follow-up to #67, not a prerequisite for opt-in eager planning.

Requested in review: https://github.com/apache/datafusion-iceberg/pull/67#discussion_r4348219778

Dominant language
Rust
Stars
30
Forks
30
Avg merge
3d 19h
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-iceberg

All issues in apache/datafusion-iceberg

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.