Implement Spark-compatible `try_add`
Mantenedores costumam responder em até 1 dia
Ninguém assumiu esta issue ainda.
Avaliação
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Facilidade para iniciantes
- 22/100
Direção de pesquisa
Start in crates/spark/src/function/math/ and read the existing date_add, make_dt_interval and try_sum implementations to see how day-time intervals map to Duration(Microsecond) and year-month ones to Interval(YearMonth). Then enable the commented queries in crates/sqllogictest/test_files/math/try_add.slt and extend them with the NULL, overflow and DST cases listed here. Done means that slt file passes and the function is registered and documented with #[user_doc].
Escrita pelo modelo de indexação a partir do texto da issue.
Descrição
Part of #174 and apache/datafusion-comet#7300.
What the function does
try_add(expr1, expr2) returns expr1 + expr2, but NULL instead of an error when the addition fails. It accepts the same operand types as +: numeric, DATE with an integer day count, DATE / TIMESTAMP / TIMESTAMP_NTZ with an interval, and interval with interval. Available since 3.2.0.
It is RuntimeReplaceable. For two numeric operands it becomes Add(l, r, EvalMode.TRY): normal type coercion (wider integral type, decimal precision rules, string to numeric), and an integer or decimal overflow returns NULL. For any other operand types it becomes TryEval(Add(l, r, EvalMode.ANSI)), which the analyzer resolves to the datetime expression for that combination (DateAdd, DateAddYMInterval, TimestampAddYMInterval, TimestampAddInterval, DateAddInterval, or interval Add), and TryEval turns any exception into NULL. The result type is the type of the resolved +.
Examples
SELECT try_add(1, 2);
-- 3
SELECT try_add(2147483647, 1);
-- NULL
SELECT try_add(2147483647, decimal(1));
-- 2147483648 decimal(11,0)
SELECT try_add(-9223372036854775808L, -1);
-- NULL
SELECT try_add(date'2021-01-01', 1);
-- 2021-01-02
SELECT try_add(date'2021-01-01', interval 1 year);
-- 2022-01-01
SELECT try_add(date'2021-01-01', interval 2 second);
-- 2021-01-01 00:00:02 timestamp
SELECT try_add(timestamp_ntz'2021-01-01 00:00:00', interval 2 second);
-- 2021-01-01 00:00:02 timestamp_ntz
SELECT try_add(interval 1 year, interval 2 year);
-- 3-0
SELECT try_add(interval 2 second, interval 2 second);
-- 0 00:00:04.000000000
SELECT try_add(interval 2147483647 month, interval 2 month);
-- NULL
SELECT try_add(interval 106751991 day, interval 3 day);
-- NULL
SELECT try_add(interval 2 year, interval 2 second);
-- error DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
Edge cases to test
- NULL in either argument, and an untyped NULL with an interval or datetime operand (the analyzer casts it to the other side's interval type; in 4.1+ a NULL next to a datetime is cast to
INTERVAL DAY TO SECOND). - Numeric: TINYINT/SMALLINT/INT/BIGINT overflow in both directions returns NULL; float/double overflow gives +/-Infinity (no NULL); NaN propagates; DECIMAL result precision/scale follows Spark's
Addrules and overflow beyond precision 38 returns NULL; string operands are coerced (try_add(2147483647, "1")is BIGINT2147483648in 4.x, DOUBLE2.147483648E9in 3.5). - Errors in the operands are not suppressed on the numeric path:
try_add(1, (2147483647 + 1))fails withARITHMETIC_OVERFLOWandtry_add(1, 1.0 / 0.0)withDIVIDE_BY_ZEROunder ANSI. On the datetime pathTryEvalwraps the wholeAdd, including its operands, so an error inside either operand becomes NULL (derived fromTryEval.eval). A UDF receives already-evaluated arguments and cannot reproduce this; document it. - DATE + integer: only INT, SMALLINT, TINYINT day counts are accepted (
DateAddusesExpectsInputTypes); the addition is plainintaddition, so it wraps rather than returning NULL (existingdate_addin this crate does the same). - DATE +
INTERVAL DAYreturns DATE; DATE + day-time interval with a time part returns TIMESTAMP (the date is cast to TIMESTAMP in the session time zone first). - DATE / TIMESTAMP + year-month interval uses
plusMonthswith end-of-month clamping (2021-01-31+ 1 month is2021-02-28). - TIMESTAMP + day-time interval adds days in
spark.sql.session.timeZonelocal time and then microseconds, soINTERVAL 1 DAYacross a DST transition adds 23 or 25 hours; TIMESTAMP_NTZ uses UTC. Test withAmerica/Los_Angelesaround 2021-03-14 and 2021-11-07. - Interval + interval: same interval type only; overflow returns NULL (examples above). Year-month and day-time intervals cannot be mixed.
- A result outside the TIMESTAMP range overflows in
instantToMicros(Math.multiplyExact/addExact) and becomes NULL. - Legacy
CalendarIntervalTypeoperands (e.g. frommake_interval):DateAddIntervalis created withansiEnabled = true, so DATE + an interval with a non-zero microseconds part throws and becomes NULL. - TIME + day-time interval (4.1+,
TimeAddInterval). - Operand order: interval or integer on the left gives the same result (
try_add(1, date'2021-01-01')is2021-01-02). spark.sql.ansi.enableddoes not changetry_additself, but changes coercion of strings and errors raised by its operands.
Differences across Spark versions
- 4.0: ANSI on by default, so operand errors such as
2147483647 + 1insidetry_addnow fail (they returned-2147483647in 3.x), and string operands coerce to BIGINT instead of DOUBLE. - 4.1:
TimeAddwas renamedTimestampAddInterval(no semantic change found); TIME type support and NULL-with-datetime casting added inBinaryArithmeticWithDatetimeResolver. TryAdditself is unchanged from 3.4 to 4.2.
Implementation notes
- Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala(TryAdd,TryEval),arithmetic.scala(Add,BinaryArithmetic.evalforEvalMode.TRY),analysis/BinaryArithmeticWithDatetimeResolver.scala(4.0+; insideAnalyzer.scalain 3.x),datetimeExpressions.scala(TimestampAddInterval,DateAddInterval,DateAddYMInterval,TimestampAddYMInterval),util/DateTimeUtils.scala. - DataFusion core has no
try_add. Its+operator wraps on integer overflow, or errors whenfail_on_overflowis set; there is no NULL-on-overflow mode. datafusion-sparkhastry_sum(aggregate),date_add,add_months,make_interval/make_dt_interval; reuse their date and interval arithmetic. Spark day-time intervals map toDuration(Microsecond)here (seemake_dt_interval.rs), year-month intervals toInterval(YearMonth).- Belongs in
crates/spark/src/function/math/. The placeholdercrates/sqllogictest/test_files/math/try_add.sltalready has commented-out queries; enable and extend it. Document with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
- Linguagem predominante
- Rust
- Estrelas
- 7
- Forks
- 15
- Merge médio
- 16h 43min
- PRs com merge (30d)
- 26
Preparar o ambiente
- Sem Dockerfile nem arquivo Docker Compose
- Sem modelo de pull request
- Ler o guia de contribuição
Primeiros passos
- Leia a issue inteira e depois o guia de contribuição do projeto.
- Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
- Faça um fork do repositório e trabalhe em uma branch.
- Abra um pull request que referencie o número da issue.
Mais de apache/datafusion-spark
-
enhancement
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 70/100
apache/datafusion-spark#287 ·
Mantenedores costumam responder em até 1 dia
-
enhancement good first issue
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 80/100
apache/datafusion-spark#266 ·
Mantenedores costumam responder em até 1 dia
-
[Bug] make_dt_interval returns NULL for more than about 1.49 million days, Spark returns the intervalTalvez já em andamento @Subodh-17 assumiu hoje. Aberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 78/100
apache/datafusion-spark#109 ·
Mantenedores costumam responder em até 1 dia
-
enhancement
Dificuldade 5/5 Mais de uma semana Facilidade para iniciantes 18/100
apache/datafusion-spark#299 ·
Mantenedores costumam responder em até 1 dia
-
enhancement
Dificuldade 4/5 3-5 dias Facilidade para iniciantes 35/100
apache/datafusion-spark#298 ·
Mantenedores costumam responder em até 1 dia
Todas as issues de apache/datafusion-spark
Issues semelhantes
-
comp-rbac
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 68/100
ClickHouse/ClickHouse#131875 ·
Mantenedores costumam responder em até 1 dia
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesTalvez já em andamento @Borg93 assumiu hoje. Abertabug
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 75/100
lance-format/lance#11117 ·
Mantenedores costumam responder em até 2 dias
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 70/100
storytold/photocraft#15232 · 1 comentário ·
Mantenedores costumam responder em até 1 dia
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Talvez já em andamento @Ynyesto assumiu hoje. Aberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 72/100
foundry-rs/foundry#18485 ·
Mantenedores costumam responder em até 1 dia
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 70/100
bondybondbond/rushcut#238 · 1 comentário ·