feat: per-element predicates on list<struct> in filters (lambda support for array_any_match)

オープン
#10,873 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python, rust, sql
領域
backend, databases

調査の方向性

Start with rust/lance-datafusion/src/sql.rs and the filter parsing and Substrait tests. The issue’s update says the filter-only syntax is being handled separately in #10901, while general lambda support depends on DataFusion changes. Check that plan and the DataFusion version in use before choosing a track; done means the selected lambda form parses and passes the listed edge case, index, and Substrait tests.

索引モデルが issue の本文から書いたものです。

説明

feature

Update: plan (Nov 10)

We're splitting this into two tracks.

  1. Lambda syntax for per-element filters, rewritten to any_element. #10901 adds any_element(list, 'predicate'). Lance would accept array_any_match(list, d -> <predicate over d's fields>) and rewrite it to any_element while parsing the filter.

    • The lambda never reaches DataFusion's lambda planning, Substrait or execution, so none of the gaps below apply.
    • Lance's own literal handling applies inside the predicate.
    • Other lambda shapes get a clear error.
  2. General lambda support waits on DataFusion. That covers other higher-order functions, and lambdas outside filters. The upstream work:

    • apache/datafusion#27088: field access on lambda parameters;
    • apache/datafusion#25960: lambdas with the default dialect;
    • the Substrait parameter-name fix in apache/datafusion#27086, merged but not yet released.

    The prototype hit two more gaps, and both are already fixed on DataFusion main: get_field now keeps a struct's nulls (apache/datafusion#25122), and a lambda parameter may shadow a column. They arrive with DataFusion 56. The original proposal below is the prototype for this track.


Summary

Filters can't express a predicate that must hold on the same element of a list<struct> column, such as "any detection with label = 'car' AND confidence > 0.9". DataFusion already ships higher-order functions for this (array_any_match, array_filter, array_transform, array_first). Lance's filter dialect doesn't parse lambdas, so they can't be used:

array_any_match(detections, d -> d.label = 'car' AND d.confidence > 0.9)
-- Schema error: No field named d

LanceDialect (rust/lance-datafusion/src/sql.rs) wraps GenericDialect and doesn't override supports_lambda_functions(), which defaults to false in sqlparser 0.62. So d -> ... is read as the JSON arrow operator. Lance already rejects JSON -> access ("JSON access is not supported"), so enabling lambdas doesn't change the meaning of any filter accepted today.

Workarounds today are a separate one-row-per-element table plus a join, or fixed positions (detections[1].label, 1-based), which can't express "any element".

What a working prototype needed

We prototyped this on top of v15.0.0-beta.14 (DataFusion 55.2, sqlparser 0.62). It works end to end through the LanceDB Python API. On synthetic data with about 10 structs per list (1M and 10M rows), it returned exactly the right rows for 6 predicates, including three attributes on one element and combinations with an indexed column, and their NOT complements.

Changes:

  1. LanceDialect::supports_lambda_functions() -> true.
  2. Field access on the lambda parameter. DataFusion's sql_compound_identifier_to_expr ignores lambda scope, so d.label fails with "No field named d.label". The prototype rewrites d.a.b and d.a[1].b on in-scope lambda parameters into subscripts (d['a']), which plan as get_field on the lambda variable. This works around an upstream gap.
  3. Substrait, encode side. normalize_expr rejects RexType::Lambda. The prototype walks the lambda body and skips LambdaParameterReference when remapping fields; LambdaInvocation stays rejected.
  4. Substrait, decode side. Substrait types carry no names, so lambda parameters decode as structs without field names, and planning fails with "Field label not found in struct". The prototype derives the named lambda parameters from the higher-order function's decoded value arguments, as the SQL planner does.

No change was needed in physical planning or higher-order function registration. The scalar-index planner treats the call as non-sargable, so it becomes the refine filter. With AND width > 1920, the width BTree still prefilters and the lambda refines.

Tests in the prototype, all passing:

  • lance-datafusion with --features substrait, including 7 Substrait round-trips;
  • lance-index expression tests;
  • a new dataset test covering scan with and without an index, count_rows, delete and update_where with a lambda predicate, and Substrait;
  • the existing scanner and filtered-read suites.

Proposal

Enable lambda parsing in the filter dialect and land the fixes above, with tests for:

  • the "same element" trap: a low-confidence car next to a high-confidence person must not match;
  • null and empty lists, null elements, a predicate that evaluates to null;
  • NOT array_any_match(...);
  • combination with indexed columns.

Upstream DataFusion issues for gaps 2 and 4 would let the workarounds go.

We have the prototype as a patch and can open a PR if this direction is welcome.

Related: #1569 (indexing fields inside complex types). An index on list elements would plan against this syntax, or against a plain function form.

主要言語
Rust
スター
7.3k
フォーク
929
平均マージ
2日 22時間
マージ済み PR(30日)
572

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

lance-format/lance のほかの issue

lance-format/lance の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。