Implement Spark-compatible `try_add`
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 22/100
Direzione di ricerca
Start in crates/spark/src/function/math/ and read the existing date_add, make_dt_interval and try_sum implementations to see how day-time intervals map to Duration(Microsecond) and year-month ones to Interval(YearMonth). Then enable the commented queries in crates/sqllogictest/test_files/math/try_add.slt and extend them with the NULL, overflow and DST cases listed here. Done means that slt file passes and the function is registered and documented with #[user_doc].
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Part of #174 and apache/datafusion-comet#7300.
What the function does
try_add(expr1, expr2) returns expr1 + expr2, but NULL instead of an error when the addition fails. It accepts the same operand types as +: numeric, DATE with an integer day count, DATE / TIMESTAMP / TIMESTAMP_NTZ with an interval, and interval with interval. Available since 3.2.0.
It is RuntimeReplaceable. For two numeric operands it becomes Add(l, r, EvalMode.TRY): normal type coercion (wider integral type, decimal precision rules, string to numeric), and an integer or decimal overflow returns NULL. For any other operand types it becomes TryEval(Add(l, r, EvalMode.ANSI)), which the analyzer resolves to the datetime expression for that combination (DateAdd, DateAddYMInterval, TimestampAddYMInterval, TimestampAddInterval, DateAddInterval, or interval Add), and TryEval turns any exception into NULL. The result type is the type of the resolved +.
Examples
SELECT try_add(1, 2);
-- 3
SELECT try_add(2147483647, 1);
-- NULL
SELECT try_add(2147483647, decimal(1));
-- 2147483648 decimal(11,0)
SELECT try_add(-9223372036854775808L, -1);
-- NULL
SELECT try_add(date'2021-01-01', 1);
-- 2021-01-02
SELECT try_add(date'2021-01-01', interval 1 year);
-- 2022-01-01
SELECT try_add(date'2021-01-01', interval 2 second);
-- 2021-01-01 00:00:02 timestamp
SELECT try_add(timestamp_ntz'2021-01-01 00:00:00', interval 2 second);
-- 2021-01-01 00:00:02 timestamp_ntz
SELECT try_add(interval 1 year, interval 2 year);
-- 3-0
SELECT try_add(interval 2 second, interval 2 second);
-- 0 00:00:04.000000000
SELECT try_add(interval 2147483647 month, interval 2 month);
-- NULL
SELECT try_add(interval 106751991 day, interval 3 day);
-- NULL
SELECT try_add(interval 2 year, interval 2 second);
-- error DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
Edge cases to test
- NULL in either argument, and an untyped NULL with an interval or datetime operand (the analyzer casts it to the other side's interval type; in 4.1+ a NULL next to a datetime is cast to
INTERVAL DAY TO SECOND). - Numeric: TINYINT/SMALLINT/INT/BIGINT overflow in both directions returns NULL; float/double overflow gives +/-Infinity (no NULL); NaN propagates; DECIMAL result precision/scale follows Spark's
Addrules and overflow beyond precision 38 returns NULL; string operands are coerced (try_add(2147483647, "1")is BIGINT2147483648in 4.x, DOUBLE2.147483648E9in 3.5). - Errors in the operands are not suppressed on the numeric path:
try_add(1, (2147483647 + 1))fails withARITHMETIC_OVERFLOWandtry_add(1, 1.0 / 0.0)withDIVIDE_BY_ZEROunder ANSI. On the datetime pathTryEvalwraps the wholeAdd, including its operands, so an error inside either operand becomes NULL (derived fromTryEval.eval). A UDF receives already-evaluated arguments and cannot reproduce this; document it. - DATE + integer: only INT, SMALLINT, TINYINT day counts are accepted (
DateAddusesExpectsInputTypes); the addition is plainintaddition, so it wraps rather than returning NULL (existingdate_addin this crate does the same). - DATE +
INTERVAL DAYreturns DATE; DATE + day-time interval with a time part returns TIMESTAMP (the date is cast to TIMESTAMP in the session time zone first). - DATE / TIMESTAMP + year-month interval uses
plusMonthswith end-of-month clamping (2021-01-31+ 1 month is2021-02-28). - TIMESTAMP + day-time interval adds days in
spark.sql.session.timeZonelocal time and then microseconds, soINTERVAL 1 DAYacross a DST transition adds 23 or 25 hours; TIMESTAMP_NTZ uses UTC. Test withAmerica/Los_Angelesaround 2021-03-14 and 2021-11-07. - Interval + interval: same interval type only; overflow returns NULL (examples above). Year-month and day-time intervals cannot be mixed.
- A result outside the TIMESTAMP range overflows in
instantToMicros(Math.multiplyExact/addExact) and becomes NULL. - Legacy
CalendarIntervalTypeoperands (e.g. frommake_interval):DateAddIntervalis created withansiEnabled = true, so DATE + an interval with a non-zero microseconds part throws and becomes NULL. - TIME + day-time interval (4.1+,
TimeAddInterval). - Operand order: interval or integer on the left gives the same result (
try_add(1, date'2021-01-01')is2021-01-02). spark.sql.ansi.enableddoes not changetry_additself, but changes coercion of strings and errors raised by its operands.
Differences across Spark versions
- 4.0: ANSI on by default, so operand errors such as
2147483647 + 1insidetry_addnow fail (they returned-2147483647in 3.x), and string operands coerce to BIGINT instead of DOUBLE. - 4.1:
TimeAddwas renamedTimestampAddInterval(no semantic change found); TIME type support and NULL-with-datetime casting added inBinaryArithmeticWithDatetimeResolver. TryAdditself is unchanged from 3.4 to 4.2.
Implementation notes
- Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala(TryAdd,TryEval),arithmetic.scala(Add,BinaryArithmetic.evalforEvalMode.TRY),analysis/BinaryArithmeticWithDatetimeResolver.scala(4.0+; insideAnalyzer.scalain 3.x),datetimeExpressions.scala(TimestampAddInterval,DateAddInterval,DateAddYMInterval,TimestampAddYMInterval),util/DateTimeUtils.scala. - DataFusion core has no
try_add. Its+operator wraps on integer overflow, or errors whenfail_on_overflowis set; there is no NULL-on-overflow mode. datafusion-sparkhastry_sum(aggregate),date_add,add_months,make_interval/make_dt_interval; reuse their date and interval arithmetic. Spark day-time intervals map toDuration(Microsecond)here (seemake_dt_interval.rs), year-month intervals toInterval(YearMonth).- Belongs in
crates/spark/src/function/math/. The placeholdercrates/sqllogictest/test_files/math/try_add.sltalready has commented-out queries; enable and extend it. Document with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
- Lingua principale
- Rust
- Stelle
- 7
- Fork
- 15
- Merge medio
- 16h 43m
- PR unite (30g)
- 26
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di apache/datafusion-spark
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
apache/datafusion-spark#287 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
apache/datafusion-spark#266 ·
I maintainer di solito rispondono entro 1 giorno
-
[Bug] make_dt_interval returns NULL for more than about 1.49 million days, Spark returns the intervalForse già presa @Subodh-17 l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
apache/datafusion-spark#109 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 18/100
apache/datafusion-spark#299 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
apache/datafusion-spark#298 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di apache/datafusion-spark
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
suhina-daw/suhina-daw#1477 ·
I maintainer di solito rispondono entro 1 giorno
-
parity priority: low ui
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
storytold/pdfcraft#7483 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
IngvarConsulting/unica#3097 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement in-progress
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 64/100