Implement Spark-compatible `forall` (higher-order array predicate)

Đang mở
  • Có gì không ổn?
#249 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
46/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust
Lĩnh vực
databases

Hướng nghiên cứu

Read datafusion/functions-nested/src/array_any_match.rs and the HigherOrderUDFImpl trait in datafusion/expr/src/higher_order_function.rs first, alongside Spark's ArrayForAll. crates/spark/src/function/lambda/mod.rs is an empty placeholder needing a HigherOrderUDF and register_higher_order_function wiring, with the pub(crate) lambda_utils.rs helpers ported locally. Reuse any-match's flatten-and-reduce shape but with a false-wins-then-NULL reduction. Done when crates/sqllogictest/test_files/lambda/forall.slt passes the NULL-vs-false ordering, empty array, nested and lambda-arity cases.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #176 and apache/datafusion-comet#7300.

What the function does

forall(expr, pred) returns whether pred holds for every element of the array expr. pred must be a
one-argument lambda returning BOOLEAN. A non-boolean body, or a non-lambda literal such as
forall(array(1, 2, 3), 1), fails with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE. There is no index argument. The
result type is BOOLEAN, nullable when argument.nullable || function.nullable. Available since Spark 3.0.0. No
aliases.

Semantics (ArrayForAll; unlike exists, there is no legacy switch, and three-valued logic always applies):

  • A NULL array gives NULL.
  • Elements are visited in order, and evaluation stops at the first false.
  • If any element gives false, the result is false.
  • Otherwise, if any element gave NULL, the result is NULL.
  • Otherwise the result is true. An empty array gives true.
Examples
SELECT forall(array(1, 2, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, 4, 8), x -> x % 2 == 0);
-- true
SELECT forall(array(1, null, 3), x -> x % 2 == 0);
-- false
SELECT forall(array(2, null, 8), x -> x % 2 == 0);
-- NULL

These come from the ExpressionDescription. HigherOrderFunctionsSuite adds the following cases:

input predicate result
[2,4,8] x IS NULL OR x % 2 == 1 false
[2,4,8] NULL NULL
[1,null,3] x IS NULL OR x % 2 == 1 true
[1,null,3] NULL NULL
[2,null,8] false false
NULL any NULL
['a0','a1','a2','a3'] startswith(x, 'a') true
[null,'b','c'] startswith(x, 'a') false

Nested: transform(aai, a -> forall(a, x -> x IS NULL OR x % 2 == 1)) on [[1,3,null],null,[4,5]] gives
[true,null,false].

Edge cases to test
  • A NULL array gives NULL. An empty array gives true. Also test NULL elements, and a predicate that is NULL for
    every element (NULL result).
  • The NULL-versus-false ordering. [null, 'b', 'c'] with startswith(x, 'a') gives false: the first element gives
    NULL, the second gives false, and false wins.
  • Short-circuiting. Evaluation stops at the first false. By code reading,
    forall(array(-1, 0), x -> 1 / x > 0) returns false even with ANSI on, because 1 / 0 is never evaluated, while
    forall(array(1, 0), x -> 1 / x > 0) raises DIVIDE_BY_ZERO with ANSI on.
  • Captured outer columns and nondeterministic predicates, evaluated per element in order.
  • Element types: primitives, strings, structs, and nested arrays.
  • Lambda arity: (x, i) -> ... fails with INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH.
Differences across Spark versions
  • None found between 3.4 and 4.2. ANSI mode is on by default from 4.0, which matters for errors inside the
    predicate.
Implementation notes
  • Spark: ArrayForAll in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala.
  • Infrastructure. DataFusion's higher-order function framework (HigherOrderUDFImpl in
    datafusion/expr/src/higher_order_function.rs, and LambdaArgument::evaluate with spread_captures for captured
    columns) is on branch-56, which this crate uses. This crate's crates/spark/src/function/lambda/mod.rs is an
    empty ScalarUDF placeholder. It needs a HigherOrderUDF list and registration through
    register_higher_order_function / SessionStateBuilder::higher_order_functions.
  • DataFusion core has no forall / all_match. array_any_match (datafusion/functions-nested/src/array_any_match.rs)
    is the mirror image and shows the framework pattern: evaluate the predicate over the flattened values, then reduce
    each row's range. A Spark forall can use the same reduction (any false gives false, else any NULL gives NULL,
    else true). Like array_any_match, though, vectorized evaluation does not short-circuit, so it raises errors and
    calls volatile functions on elements Spark never visits. Handle that the same way as the planned exists.
  • The lambda_utils.rs helpers in functions-nested are pub(crate), so equivalents have to live in this crate
    (they could be shared with exists).
  • Add crates/sqllogictest/test_files/lambda/forall.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.