Implement Spark-compatible `array_sort`

Aperta
  • Qualcosa non va?
#269 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
64/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend

Direzione di ricerca

Start by reading native/spark-expr/src/array_funcs/sort_array.rs, which already holds the float-ordering and stable-sort logic to share. Then add a SparkArraySort expression in crates/spark/src/function/array/, register it as spark_array_sort, and document it with #[user_doc]. Write crates/sqllogictest/test_files/array/array_sort.slt covering the NULL-last, NaN, signed-zero and stability cases listed above; done means those cases match Spark 3.4-4.1 output. Leave the lambda form for later.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #178. Split out of #145, which was closed as a duplicate of #218 (sort_array).

What the function does

array_sort(array[, func]) sorts an array. Without func it uses Spark's default comparator, which sorts ascending and
puts NULL elements last. This differs from sort_array, which puts NULLs first when ascending. The result type is the
input type. With func, the lambda (left, right) -> int is the comparator. It must return INT, or analysis fails with
DATATYPE_MISMATCH.UNEXPECTED_RETURN_TYPE. If it returns NULL, the query fails unless
spark.sql.legacy.allowNullComparisonResultInArraySort (internal, default false) is set, in which case NULL counts as 0.

Spark implements this as ArraySort in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:

  • It is a higher-order function even without func. ArraySort.defaultComparator is the lambda
    if(left IS NULL AND right IS NULL, 0, if(left IS NULL, 1, if(right IS NULL, -1, if(left < right, -1, if(left > right, 1, 0))))).
  • Only top-level NULL elements are placed last. Inside struct and array elements, < and > order NULLs below non-null values.
  • Floats compare with < and >, so NaN is greater than every non-NaN value, all NaNs are equal, and -0.0 equals 0.0.
  • It sorts with java.util.Arrays.sort(Object[], Comparator), which is stable, so equal elements keep their input order.
    An array of NullType elements is returned unchanged.
Examples
SELECT array_sort(array('b', 'd', null, 'c', 'a'));
-- ["a","b","c","d",null]
SELECT array_sort(array(5, 6, 1), (left, right) -> case when left < right then -1 when left > right then 1 else 0 end);
-- [1,5,6]
SELECT array_sort(array(3, 1, 2), 1);
-- [3,1,2]   (a constant comparator; the stable sort keeps the input order)
SELECT array_sort(array(3, 1, 2), map('compare', 0));
-- error DATATYPE_MISMATCH.UNEXPECTED_RETURN_TYPE
Comet implementation

Comet implements the default-comparator form (SparkArraySort, registered as spark_array_sort). It shares the
float-ordering and stable-sort code with sort_array:

Edge cases to test
  • NULL array, empty array, an array of only NULLs, and array(NULL, NULL) (element type VOID).
  • NULLs at the start, middle and end of the input. All of them go last.
  • Struct and array elements that hold NULLs. Those NULLs sort first within the element ordering.
  • NaN (with different bit patterns), ±Infinity and signed zeros. [0.0, -0.0] and [-0.0, 0.0] keep their order.
  • Stability for equal elements, for example structs whose fields compare equal.
Differences across Spark versions
  • 3.4 and 3.5 report paramIndex "1" for a non-array argument; 4.0 and 4.1 report it as an ordinal ("first").
  • No other differences in ArraySort between 3.4.3, 3.5.8, 4.0.1 and 4.1.1.
Implementation notes
  • The lambda form depends on lambda support. The higher-order functions in #176 have the same dependency, so start with
    the default comparator.
  • DataFusion core's array_sort uses an unstable sort and IEEE 754 total order for floats, as #218 describes.
    It can be used with 'ASC', 'NULLS LAST' only for non-float element types with no nested NULLs, if the stability
    difference is acceptable there.
  • The function belongs in crates/spark/src/function/array/. Add crates/sqllogictest/test_files/array/array_sort.slt
    and document it with #[user_doc].

This issue was generated by an LLM (Claude Code) from the Apache Spark source code, Spark's .sql.out golden files and
Comet's source. Please verify the details against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.