Expose Parquet read-ahead controls for Iceberg SQL scans

Open
  • Is something wrong?
#58 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
51/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
backend, databases

Research direction

Read IcebergDataSource and inspect how it builds TableScan for static and catalog-backed providers; first check whether apache/iceberg-rust#3395 has landed and how the pinned revision can be updated. Trace the existing storage range tests and run them with page pruning disabled, including a merged request larger than the configured gap. Done means optional settings reach both provider types, unset settings preserve current behavior, and the tests confirm the requested byte ranges and unchanged query results.

Written by the indexing model from the issue text.

Description

IcebergDataSource currently builds TableScan without forwarding read-ahead configuration. apache/iceberg-rust#3395 adds the underlying TableScanBuilder methods.

Expose optional footer metadata size hints and range-coalescing gap settings through the DataFusion integration. Preserve existing behavior when unset, and support both static and catalog-backed table providers.

Tests should verify actual storage byte ranges and unchanged query results with page pruning disabled, including a merged request larger than the configured gap.

This proposes configurable controls without changing defaults. It depends on apache/iceberg-rust#3395 landing and updating the integration’s pinned Iceberg revision. I’m willing to implement this.

Dominant language
Rust
Stars
30
Forks
30
Avg merge
3d 19h
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-iceberg

All issues in apache/datafusion-iceberg

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.