Implement Spark-compatible `forall` (higher-order array predicate)

オープン
  • 何か問題がありますか？
#249 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
46/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
rust
領域
databases

調査の方向性

Read datafusion/functions-nested/src/array_any_match.rs and the HigherOrderUDFImpl trait in datafusion/expr/src/higher_order_function.rs first, alongside Spark's ArrayForAll. crates/spark/src/function/lambda/mod.rs is an empty placeholder needing a HigherOrderUDF and register_higher_order_function wiring, with the pub(crate) lambda_utils.rs helpers ported locally. Reuse any-match's flatten-and-reduce shape but with a false-wins-then-NULL reduction. Done when crates/sqllogictest/test_files/lambda/forall.slt passes the NULL-vs-false ordering, empty array, nested and lambda-arity cases.

索引モデルが issue の本文から書いたものです。

説明

enhancement

Part of #176 and apache/datafusion-comet#7300.

What the function does

forall(expr, pred) returns whether pred holds for every element of the array expr. pred must be a
one-argument lambda returning BOOLEAN. A non-boolean body, or a non-lambda literal such as
forall(array(1, 2, 3), 1), fails with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE. There is no index argument. The
result type is BOOLEAN, nullable when argument.nullable || function.nullable. Available since Spark 3.0.0. No
aliases.

Semantics (ArrayForAll; unlike exists, there is no legacy switch, and three-valued logic always applies):

  • A NULL array gives NULL.
  • Elements are visited in order, and evaluation stops at the first false.
  • If any element gives false, the result is false.
  • Otherwise, if any element gave NULL, the result is NULL.
  • Otherwise the result is true. An empty array gives true.
Examples
SELECT forall(array(1, 2, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, 4, 8), x -> x % 2 == 0);
-- true
SELECT forall(array(1, null, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, null, 8), x -> x % 2 == 0);
-- NULL

These come from the ExpressionDescription. HigherOrderFunctionsSuite adds the following cases:

input predicate result
[2,4,8] x IS NULL OR x % 2 == 1 false
[2,4,8] NULL NULL
[1,null,3] x IS NULL OR x % 2 == 1 true
[1,null,3] NULL NULL
[2,null,8] false false
NULL any NULL
['a0','a1','a2','a3'] startswith(x, 'a') true
[null,'b','c'] startswith(x, 'a') false

Nested: transform(aai, a -> forall(a, x -> x IS NULL OR x % 2 == 1)) on [[1,3,null],null,[4,5]] gives
[true,null,false].

Edge cases to test
  • A NULL array gives NULL. An empty array gives true. Also test NULL elements, and a predicate that is NULL for
    every element (NULL result).
  • The NULL-versus-false ordering. [null, 'b', 'c'] with startswith(x, 'a') gives false: the first element gives
    NULL, the second gives false, and false wins.
  • Short-circuiting. Evaluation stops at the first false. By code reading,
    forall(array(-1, 0), x -> 1 / x > 0) returns false even with ANSI on, because 1 / 0 is never evaluated, while
    forall(array(1, 0), x -> 1 / x > 0) raises DIVIDE_BY_ZERO with ANSI on.
  • Captured outer columns and nondeterministic predicates, evaluated per element in order.
  • Element types: primitives, strings, structs, and nested arrays.
  • Lambda arity: (x, i) -> ... fails with INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH.
Differences across Spark versions
  • None found between 3.4 and 4.2. ANSI mode is on by default from 4.0, which matters for errors inside the
    predicate.
Implementation notes
  • Spark: ArrayForAll in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala.
  • Infrastructure. DataFusion's higher-order function framework (HigherOrderUDFImpl in
    datafusion/expr/src/higher_order_function.rs, and LambdaArgument::evaluate with spread_captures for captured
    columns) is on branch-56, which this crate uses. This crate's crates/spark/src/function/lambda/mod.rs is an
    empty ScalarUDF placeholder. It needs a HigherOrderUDF list and registration through
    register_higher_order_function / SessionStateBuilder::higher_order_functions.
  • DataFusion core has no forall / all_match. array_any_match (datafusion/functions-nested/src/array_any_match.rs)
    is the mirror image and shows the framework pattern: evaluate the predicate over the flattened values, then reduce
    each row's range. A Spark forall can use the same reduction (any false gives false, else any NULL gives NULL,
    else true). Like array_any_match, though, vectorized evaluation does not short-circuit, so it raises errors and
    calls volatile functions on elements Spark never visits. Handle that the same way as the planned exists.
  • The lambda_utils.rs helpers in functions-nested are pub(crate), so equivalents have to live in this crate
    (they could be shared with exists).
  • Add crates/sqllogictest/test_files/lambda/forall.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

主要言語
Rust
スター
7
フォーク
15
平均マージ
16時間 43分
マージ済み PR（30日）
26

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion-spark のほかの issue

apache/datafusion-spark の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。