Implement Spark-compatible `transform_values` (higher-order map function)

Aperta
  • Qualcosa non va?
#254 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, sql
Ambito
databases

Direzione di ricerca

Start by reading datafusion/functions-nested/src/array_transform.rs as the model for evaluating a lambda over flattened entries, then look at crates/spark/src/function/lambda/mod.rs, which is still an empty ScalarUDF placeholder needing a HigherOrderUDF list and registration. Reuse the MapArray helpers in crates/spark/src/function/map/utils.rs to rebuild the output map with the original offsets, nulls and key array. Done means crates/sqllogictest/test_files/lambda/transform_values.slt passes against the examples and edge cases above, including arity mismatch errors and NULL/empty maps.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #176 and apache/datafusion-comet#7300.

What the function does

transform_values(expr, func) returns a map with the same keys, in the same order, whose values are func(k, v)
for each entry of the map expr. The result type is MAP<keyType, func.dataType> with
valueContainsNull = func.nullable. A NULL map gives NULL, and the result is nullable if the input is. The lambda
takes exactly 2 parameters: k (non-null) and v (nullable when the input's valueContainsNull is true). expr
must be a MAP (DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE otherwise). A non-lambda func is wrapped in a hidden
lambda. Available since Spark 3.0.0. No aliases.

Keys are not modified, so there is no dedup policy and no NULL_MAP_KEY check. Spark reuses the input key array
directly (new ArrayBasedMapData(map.keyArray(), resultValues)).

Examples
SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> v + 1);
-- {1:2,2:3,3:4}
SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> k + v);
-- {1:2,2:4,3:6}
-- nested view: (1, map(1,1,2,2,3,3)), (2, map(4,4,5,5,6,6)) as t(x, ys)
SELECT transform_values(ys, (k, v) -> k + v) FROM nested;
-- {1:2,2:4,3:6} / {4:8,5:10,6:12}
SELECT transform_values(map('a', 1, 'b', 2), 999);
-- {"a":999,"b":999}
SELECT transform_values(map('a', 1, 'b', 2), struct('val', 999));
-- {"a":{"col1":"val","col2":999},"b":{"col1":"val","col2":999}}

These come from the ExpressionDescription and higher-order-functions.sql.out. HigherOrderFunctionsSuite adds:

map lambda result
{1:1, 2:null, 3:3} v + 1 {1:2, 2:null, 3:4}
{1:1, 2:null, 3:3} k * k {1:1, 2:4, 3:9}
{"a":"xy", "bb":"yz", "ccc":"zx"} length(v) + 1 {"a":3, "bb":3, "ccc":3}
{"a":"xy", "bb":null, "ccc":"zx"} concat(k, v) {"a":"axy", "bb":null, "ccc":"ccczx"}
{} any {}
NULL any NULL
Edge cases to test
  • A NULL map gives NULL. An empty map gives {}, also when the value type changes (MAP<STRING, INT> from
    length(v) + 1).
  • NULL values passed to the lambda: v + 1 gives NULL there, while k * k ignores v and gives a value. The result
    valueContainsNull follows the lambda's nullability.
  • Value-type changes (string to int, int to struct, int to map). Chained calls:
    transform_values(transform_values(m, (k, v) -> concat(k, v)), (k, v) -> concat(k, v)) on
    {"a":"xy","bb":null,"ccc":"zx"} gives {"a":"aaxy","bb":null,"ccc":"cccccczx"}.
  • Key order and key values are preserved exactly, including float keys such as -0.0 and NaN, which are not
    re-normalized. Use the input key array as it is.
  • Lambda arity: v -> v + 1 and (k, v, x) -> ... must fail with INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH.
    DataFusion's planner allows a lambda to declare fewer parameters than offered, so this check has to be explicit.
  • Captured outer columns, nondeterministic value expressions (once per entry), and ANSI errors inside the lambda
    (v / 0, overflow).
  • Sliced MapArray input and values hidden under NULL map rows. The lambda must not be evaluated on them, because
    they could raise errors Spark never raises.
Differences across Spark versions
  • None found in TransformValues between 3.4 and 4.2. The literal-argument cases in the golden file were added in
    4.1, which only adds tests. ANSI mode is on by default from 4.0.
Implementation notes
  • Spark: TransformValues in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala.
  • Infrastructure. DataFusion's higher-order function framework (HigherOrderUDFImpl in
    datafusion/expr/src/higher_order_function.rs, and LambdaArgument::evaluate with spread_captures for captured
    columns) is on branch-56, which this crate uses. lambda_parameters should offer [key (non-null), value].
  • DataFusion core has no higher-order map functions. This is the simplest of the map functions to build:
    1. Evaluate the lambda once over the flattened entry keys and values, spreading captures with the per-entry row
      index, as array_transform does in datafusion/functions-nested/src/array_transform.rs.
    2. Rebuild the MapArray with the original offsets, nulls and keys, and the new values.
  • This crate's crates/spark/src/function/lambda/mod.rs is an empty ScalarUDF placeholder. It needs a
    HigherOrderUDF list and registration through register_higher_order_function /
    SessionStateBuilder::higher_order_functions. crates/spark/src/function/map/utils.rs has MapArray helpers.
  • Add crates/sqllogictest/test_files/lambda/transform_values.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.