Support Decimal128 literals in predicate pushdown

Open
  • Is something wrong?
#73 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust, sql
Domain
data

Research direction

Start in the predicate pushdown path where scalar_value_to_datum converts DataFusion scalar values into Iceberg datums, and find where Decimal128 currently falls through as unsupported. Read the linked #59 cast-stripping discussion first, since the issue says this should merge after it. Done means exact conversion of representable Decimal128 literals, with tests for comparisons, IN lists, negatives, NULLs, precision boundaries, and casts that are left to DataFusion.

Written by the indexing model from the issue text.

Description

Hi there,

Decimal128 literals are currently unsupported by
scalar_value_to_datum, so decimal comparisons are omitted from
the predicate passed to Iceberg.

For a price DECIMAL(18,2) column, for example:

SELECT id FROM prices
WHERE price > CAST('12.34' AS DECIMAL(18,2));

DataFusion evaluates the filter, but Iceberg cannot use that
decimal predicate for pruning.

Support exact conversion of representable Decimal128 literals,
preserving their value, precision, and scale. Leave unsupported
literals to DataFusion.

Cover comparisons and IN predicates, negative values, NULLs,
precision boundaries, and conservative handling of casts.

This should merge after #59: adding decimal conversion exposes
the existing cast-stripping problem for comparisons that require
type coercion.

I have a draft implementation and would be happy to contribute it

Dominant language
Rust
Stars
30
Forks
30
Avg merge
3d 19h
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-iceberg

All issues in apache/datafusion-iceberg

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.