Implement Spark-compatible `forall` (higher-order array predicate)

Aperta
  • Qualcosa non va?
#249 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
46/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
databases

Direzione di ricerca

Read datafusion/functions-nested/src/array_any_match.rs and the HigherOrderUDFImpl trait in datafusion/expr/src/higher_order_function.rs first, alongside Spark's ArrayForAll. crates/spark/src/function/lambda/mod.rs is an empty placeholder needing a HigherOrderUDF and register_higher_order_function wiring, with the pub(crate) lambda_utils.rs helpers ported locally. Reuse any-match's flatten-and-reduce shape but with a false-wins-then-NULL reduction. Done when crates/sqllogictest/test_files/lambda/forall.slt passes the NULL-vs-false ordering, empty array, nested and lambda-arity cases.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #176 and apache/datafusion-comet#7300.

What the function does

forall(expr, pred) returns whether pred holds for every element of the array expr. pred must be a
one-argument lambda returning BOOLEAN. A non-boolean body, or a non-lambda literal such as
forall(array(1, 2, 3), 1), fails with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE. There is no index argument. The
result type is BOOLEAN, nullable when argument.nullable || function.nullable. Available since Spark 3.0.0. No
aliases.

Semantics (ArrayForAll; unlike exists, there is no legacy switch, and three-valued logic always applies):

  • A NULL array gives NULL.
  • Elements are visited in order, and evaluation stops at the first false.
  • If any element gives false, the result is false.
  • Otherwise, if any element gave NULL, the result is NULL.
  • Otherwise the result is true. An empty array gives true.
Examples
SELECT forall(array(1, 2, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, 4, 8), x -> x % 2 == 0);
-- true
SELECT forall(array(1, null, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, null, 8), x -> x % 2 == 0);
-- NULL

These come from the ExpressionDescription. HigherOrderFunctionsSuite adds the following cases:

input predicate result
[2,4,8] x IS NULL OR x % 2 == 1 false
[2,4,8] NULL NULL
[1,null,3] x IS NULL OR x % 2 == 1 true
[1,null,3] NULL NULL
[2,null,8] false false
NULL any NULL
['a0','a1','a2','a3'] startswith(x, 'a') true
[null,'b','c'] startswith(x, 'a') false

Nested: transform(aai, a -> forall(a, x -> x IS NULL OR x % 2 == 1)) on [[1,3,null],null,[4,5]] gives
[true,null,false].

Edge cases to test
  • A NULL array gives NULL. An empty array gives true. Also test NULL elements, and a predicate that is NULL for
    every element (NULL result).
  • The NULL-versus-false ordering. [null, 'b', 'c'] with startswith(x, 'a') gives false: the first element gives
    NULL, the second gives false, and false wins.
  • Short-circuiting. Evaluation stops at the first false. By code reading,
    forall(array(-1, 0), x -> 1 / x > 0) returns false even with ANSI on, because 1 / 0 is never evaluated, while
    forall(array(1, 0), x -> 1 / x > 0) raises DIVIDE_BY_ZERO with ANSI on.
  • Captured outer columns and nondeterministic predicates, evaluated per element in order.
  • Element types: primitives, strings, structs, and nested arrays.
  • Lambda arity: (x, i) -> ... fails with INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH.
Differences across Spark versions
  • None found between 3.4 and 4.2. ANSI mode is on by default from 4.0, which matters for errors inside the
    predicate.
Implementation notes
  • Spark: ArrayForAll in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala.
  • Infrastructure. DataFusion's higher-order function framework (HigherOrderUDFImpl in
    datafusion/expr/src/higher_order_function.rs, and LambdaArgument::evaluate with spread_captures for captured
    columns) is on branch-56, which this crate uses. This crate's crates/spark/src/function/lambda/mod.rs is an
    empty ScalarUDF placeholder. It needs a HigherOrderUDF list and registration through
    register_higher_order_function / SessionStateBuilder::higher_order_functions.
  • DataFusion core has no forall / all_match. array_any_match (datafusion/functions-nested/src/array_any_match.rs)
    is the mirror image and shows the framework pattern: evaluate the predicate over the flattened values, then reduce
    each row's range. A Spark forall can use the same reduction (any false gives false, else any NULL gives NULL,
    else true). Like array_any_match, though, vectorized evaluation does not short-circuit, so it raises errors and
    calls volatile functions on elements Spark never visits. Handle that the same way as the planned exists.
  • The lambda_utils.rs helpers in functions-nested are pub(crate), so equivalents have to live in this crate
    (they could be shared with exists).
  • Add crates/sqllogictest/test_files/lambda/forall.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.