Backport ANSI skipped-operand fix for arithmetic and concat to branch-1.2

Aperta
  • Qualcosa non va?
#7,542 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust, scala
Ambito
backend, databases

Direzione di ricerca

Read docs/source/contributor-guide/backporting.md and compare the branch-1.2 implementation using NullShortCircuit with the fix proposed in PR 7456, which depends on LazyLogical from PR 7326. Reproduce the ANSI-mode query in the issue and investigate whether wrapping operands gives Spark-compatible results and what performance tradeoff it introduces. Done means proposing and implementing a viable backport approach with correctness and performance addressed.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:expressions bug priority:medium
Describe the bug

branch-1.2 has the ANSI bug from https://github.com/apache/datafusion-comet/issues/7246 for arithmetic and concat: native Comet evaluates an operand on rows where Spark skips it, so a query Spark answers fails with CAST_INVALID_INPUT. For example, with spark.sql.ansi.enabled=true:

CREATE TABLE t(n int, txt string) USING parquet;
INSERT INTO t VALUES (NULL, 'bad'), (3, '6');
SELECT n + CAST(txt AS INT) FROM t;  -- Spark: NULL, 9. Comet: CAST_INVALID_INPUT

The fix on main is https://github.com/apache/datafusion-comet/pull/7456 (not merged yet). It cannot be cherry-picked as is: it builds on https://github.com/apache/datafusion-comet/pull/7326 (LazyLogical, speculative evaluation), which branch-1.2 does not have. branch-1.2 does have NullShortCircuit (https://github.com/apache/datafusion-comet/pull/6716), so a backport could wrap the operands without speculation.

This issue tracks whether, and how, to backport the fix to branch-1.2, as docs/source/contributor-guide/backporting.md asks for a correctness fix.

Steps to reproduce

Run the query above on branch-1.2 with ANSI mode on.

Expected behavior

Comet returns the same result as Spark: NULL where n is NULL, without evaluating CAST(txt AS INT) there.

Additional context
Willingness to contribute

I can contribute a fix for this bug independently

Lingua principale
Scala
Stelle
1.3k
Fork
414
Merge medio
3g 5h
PR unite (30g)
210

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.