Native TopK evaluates sort keys for a single-row input, raising false ANSI errors

Aperta
  • Qualcosa non va?
#7,527 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
scala, spark
Ambito
backend, performance

Direzione di ricerca

Start by reading the TopK implementation and its tests, then compare its handling of a single-row batch with Spark's TakeOrderedAndProjectExec behavior described in the issue. Reproduce the SQL example with ANSI mode enabled and adaptive execution disabled. Done means the query returns the row in Comet as it does in Spark, without raising ARITHMETIC_OVERFLOW.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug priority:medium
Describe the bug

With spark.sql.ansi.enabled=true, a TopK (ORDER BY ... LIMIT n) whose whole input is a single row fails in Comet with ARITHMETIC_OVERFLOW when its first sort key overflows on that row, while Spark returns the row.

Spark's TakeOrderedAndProjectExec goes through takeOrdered, which uses Guava's Ordering.leastOf (TopKSelector) per partition and a BoundedPriorityQueue to merge. Both store the first element without a comparison, and sorting a single element compares nothing, so Spark's lazily generated ordering never evaluates any sort key when the input has one row. Comet's native TopK evaluates every sort key over the whole batch.

With two or more rows Spark compares and evaluates the first key for every row, so both engines fail. This is pre-existing and independent of https://github.com/apache/datafusion-comet/pull/7457 , which falls back when a sort key after the first can fail; it was raised in review there.

Steps to reproduce
SET spark.sql.ansi.enabled=true;
SET spark.sql.adaptive.enabled=false;
CREATE TABLE one_row(a int, b int) USING parquet;
INSERT INTO one_row VALUES (1, 2147483647);

SELECT a, b FROM one_row ORDER BY b + 1 LIMIT 10;

Spark returns 1, 2147483647. Comet fails with [ARITHMETIC_OVERFLOW] integer overflow (checked on Spark 4.1 with a CometSqlFileTestSuite fixture, current main plus #7457).

Expected behavior

Same result as Spark. Since it only affects a single-row input, it may be enough to document it rather than fall back for every TopK with a fallible first key.

Additional context

Suggested labels: priority:low, correctness.

Lingua principale
Scala
Stelle
1.3k
Fork
414
Merge medio
3g 5h
PR unite (30g)
210

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.