Implement Spark-compatible `transform_values` (higher-order map function)

Đang mở
  • Có gì không ổn?
#254 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust, sql
Lĩnh vực
databases

Hướng nghiên cứu

Start by reading datafusion/functions-nested/src/array_transform.rs as the model for evaluating a lambda over flattened entries, then look at crates/spark/src/function/lambda/mod.rs, which is still an empty ScalarUDF placeholder needing a HigherOrderUDF list and registration. Reuse the MapArray helpers in crates/spark/src/function/map/utils.rs to rebuild the output map with the original offsets, nulls and key array. Done means crates/sqllogictest/test_files/lambda/transform_values.slt passes against the examples and edge cases above, including arity mismatch errors and NULL/empty maps.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #176 and apache/datafusion-comet#7300.

What the function does

transform_values(expr, func) returns a map with the same keys, in the same order, whose values are func(k, v)
for each entry of the map expr. The result type is MAP<keyType, func.dataType> with
valueContainsNull = func.nullable. A NULL map gives NULL, and the result is nullable if the input is. The lambda
takes exactly 2 parameters: k (non-null) and v (nullable when the input's valueContainsNull is true). expr
must be a MAP (DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE otherwise). A non-lambda func is wrapped in a hidden
lambda. Available since Spark 3.0.0. No aliases.

Keys are not modified, so there is no dedup policy and no NULL_MAP_KEY check. Spark reuses the input key array
directly (new ArrayBasedMapData(map.keyArray(), resultValues)).

Examples
SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> v + 1);
-- {1:2,2:3,3:4}
SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> k + v);
-- {1:2,2:4,3:6}
-- nested view: (1, map(1,1,2,2,3,3)), (2, map(4,4,5,5,6,6)) as t(x, ys)
SELECT transform_values(ys, (k, v) -> k + v) FROM nested;
-- {1:2,2:4,3:6} / {4:8,5:10,6:12}
SELECT transform_values(map('a', 1, 'b', 2), 999);
-- {"a":999,"b":999}
SELECT transform_values(map('a', 1, 'b', 2), struct('val', 999));
-- {"a":{"col1":"val","col2":999},"b":{"col1":"val","col2":999}}

These come from the ExpressionDescription and higher-order-functions.sql.out. HigherOrderFunctionsSuite adds:

map lambda result
{1:1, 2:null, 3:3} v + 1 {1:2, 2:null, 3:4}
{1:1, 2:null, 3:3} k * k {1:1, 2:4, 3:9}
{"a":"xy", "bb":"yz", "ccc":"zx"} length(v) + 1 {"a":3, "bb":3, "ccc":3}
{"a":"xy", "bb":null, "ccc":"zx"} concat(k, v) {"a":"axy", "bb":null, "ccc":"ccczx"}
{} any {}
NULL any NULL
Edge cases to test
  • A NULL map gives NULL. An empty map gives {}, also when the value type changes (MAP<STRING, INT> from
    length(v) + 1).
  • NULL values passed to the lambda: v + 1 gives NULL there, while k * k ignores v and gives a value. The result
    valueContainsNull follows the lambda's nullability.
  • Value-type changes (string to int, int to struct, int to map). Chained calls:
    transform_values(transform_values(m, (k, v) -> concat(k, v)), (k, v) -> concat(k, v)) on
    {"a":"xy","bb":null,"ccc":"zx"} gives {"a":"aaxy","bb":null,"ccc":"cccccczx"}.
  • Key order and key values are preserved exactly, including float keys such as -0.0 and NaN, which are not
    re-normalized. Use the input key array as it is.
  • Lambda arity: v -> v + 1 and (k, v, x) -> ... must fail with INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH.
    DataFusion's planner allows a lambda to declare fewer parameters than offered, so this check has to be explicit.
  • Captured outer columns, nondeterministic value expressions (once per entry), and ANSI errors inside the lambda
    (v / 0, overflow).
  • Sliced MapArray input and values hidden under NULL map rows. The lambda must not be evaluated on them, because
    they could raise errors Spark never raises.
Differences across Spark versions
  • None found in TransformValues between 3.4 and 4.2. The literal-argument cases in the golden file were added in
    4.1, which only adds tests. ANSI mode is on by default from 4.0.
Implementation notes
  • Spark: TransformValues in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala.
  • Infrastructure. DataFusion's higher-order function framework (HigherOrderUDFImpl in
    datafusion/expr/src/higher_order_function.rs, and LambdaArgument::evaluate with spread_captures for captured
    columns) is on branch-56, which this crate uses. lambda_parameters should offer [key (non-null), value].
  • DataFusion core has no higher-order map functions. This is the simplest of the map functions to build:
    1. Evaluate the lambda once over the flattened entry keys and values, spreading captures with the per-entry row
      index, as array_transform does in datafusion/functions-nested/src/array_transform.rs.
    2. Rebuild the MapArray with the original offsets, nulls and keys, and the new values.
  • This crate's crates/spark/src/function/lambda/mod.rs is an empty ScalarUDF placeholder. It needs a
    HigherOrderUDF list and registration through register_higher_order_function /
    SessionStateBuilder::higher_order_functions. crates/spark/src/function/map/utils.rs has MapArray helpers.
  • Add crates/sqllogictest/test_files/lambda/transform_values.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.