Implement Spark-compatible `transform_values` (higher-order map function)

Abierto
  • ¿Algo no está bien?
#254 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust, sql
Área
databases

Línea de trabajo

Start by reading datafusion/functions-nested/src/array_transform.rs as the model for evaluating a lambda over flattened entries, then look at crates/spark/src/function/lambda/mod.rs, which is still an empty ScalarUDF placeholder needing a HigherOrderUDF list and registration. Reuse the MapArray helpers in crates/spark/src/function/map/utils.rs to rebuild the output map with the original offsets, nulls and key array. Done means crates/sqllogictest/test_files/lambda/transform_values.slt passes against the examples and edge cases above, including arity mismatch errors and NULL/empty maps.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

Part of #176 and apache/datafusion-comet#7300.

What the function does

transform_values(expr, func) returns a map with the same keys, in the same order, whose values are func(k, v)
for each entry of the map expr. The result type is MAP<keyType, func.dataType> with
valueContainsNull = func.nullable. A NULL map gives NULL, and the result is nullable if the input is. The lambda
takes exactly 2 parameters: k (non-null) and v (nullable when the input's valueContainsNull is true). expr
must be a MAP (DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE otherwise). A non-lambda func is wrapped in a hidden
lambda. Available since Spark 3.0.0. No aliases.

Keys are not modified, so there is no dedup policy and no NULL_MAP_KEY check. Spark reuses the input key array
directly (new ArrayBasedMapData(map.keyArray(), resultValues)).

Examples
SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> v + 1);
-- {1:2,2:3,3:4}
SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> k + v);
-- {1:2,2:4,3:6}
-- nested view: (1, map(1,1,2,2,3,3)), (2, map(4,4,5,5,6,6)) as t(x, ys)
SELECT transform_values(ys, (k, v) -> k + v) FROM nested;
-- {1:2,2:4,3:6} / {4:8,5:10,6:12}
SELECT transform_values(map('a', 1, 'b', 2), 999);
-- {"a":999,"b":999}
SELECT transform_values(map('a', 1, 'b', 2), struct('val', 999));
-- {"a":{"col1":"val","col2":999},"b":{"col1":"val","col2":999}}

These come from the ExpressionDescription and higher-order-functions.sql.out. HigherOrderFunctionsSuite adds:

map lambda result
{1:1, 2:null, 3:3} v + 1 {1:2, 2:null, 3:4}
{1:1, 2:null, 3:3} k * k {1:1, 2:4, 3:9}
{"a":"xy", "bb":"yz", "ccc":"zx"} length(v) + 1 {"a":3, "bb":3, "ccc":3}
{"a":"xy", "bb":null, "ccc":"zx"} concat(k, v) {"a":"axy", "bb":null, "ccc":"ccczx"}
{} any {}
NULL any NULL
Edge cases to test
  • A NULL map gives NULL. An empty map gives {}, also when the value type changes (MAP<STRING, INT> from
    length(v) + 1).
  • NULL values passed to the lambda: v + 1 gives NULL there, while k * k ignores v and gives a value. The result
    valueContainsNull follows the lambda's nullability.
  • Value-type changes (string to int, int to struct, int to map). Chained calls:
    transform_values(transform_values(m, (k, v) -> concat(k, v)), (k, v) -> concat(k, v)) on
    {"a":"xy","bb":null,"ccc":"zx"} gives {"a":"aaxy","bb":null,"ccc":"cccccczx"}.
  • Key order and key values are preserved exactly, including float keys such as -0.0 and NaN, which are not
    re-normalized. Use the input key array as it is.
  • Lambda arity: v -> v + 1 and (k, v, x) -> ... must fail with INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH.
    DataFusion's planner allows a lambda to declare fewer parameters than offered, so this check has to be explicit.
  • Captured outer columns, nondeterministic value expressions (once per entry), and ANSI errors inside the lambda
    (v / 0, overflow).
  • Sliced MapArray input and values hidden under NULL map rows. The lambda must not be evaluated on them, because
    they could raise errors Spark never raises.
Differences across Spark versions
  • None found in TransformValues between 3.4 and 4.2. The literal-argument cases in the golden file were added in
    4.1, which only adds tests. ANSI mode is on by default from 4.0.
Implementation notes
  • Spark: TransformValues in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala.
  • Infrastructure. DataFusion's higher-order function framework (HigherOrderUDFImpl in
    datafusion/expr/src/higher_order_function.rs, and LambdaArgument::evaluate with spread_captures for captured
    columns) is on branch-56, which this crate uses. lambda_parameters should offer [key (non-null), value].
  • DataFusion core has no higher-order map functions. This is the simplest of the map functions to build:
    1. Evaluate the lambda once over the flattened entry keys and values, spreading captures with the per-entry row
      index, as array_transform does in datafusion/functions-nested/src/array_transform.rs.
    2. Rebuild the MapArray with the original offsets, nulls and keys, and the new values.
  • This crate's crates/spark/src/function/lambda/mod.rs is an empty ScalarUDF placeholder. It needs a
    HigherOrderUDF list and registration through register_higher_order_function /
    SessionStateBuilder::higher_order_functions. crates/spark/src/function/map/utils.rs has MapArray helpers.
  • Add crates/sqllogictest/test_files/lambda/transform_values.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lenguaje dominante
Rust
Estrellas
7
Forks
15
Merge medio
16 h 43 min
PR fusionados (30 d)
26

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-spark

Todos los issues de apache/datafusion-spark

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.