Native timestamp casts fail or panic for years beyond chrono's range

Aperta
  • Qualcosa non va?
#7,471 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
56/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, scala, spark
Ambito
backend, databases

Direzione di ricerca

Start with pre_timestamp_cast and the timestamp conversion functions in native/spark-expr/src/utils.rs, then inspect date-to-timestamp handling in native/spark-expr/src/conversion_funcs/temporal.rs and test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs. Reproduce with the SQL cases in spark/src/test/resources/sql-tests/... under the default Spark 4.1 profile. Done means out-of-range values return Spark-compatible results through fallback, including TRY casts, without native errors or panics.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

requires-triage
Describe the bug

Native casts to and from timestamps fail for values whose year is outside chrono's range (about ±262,143), while Spark supports timestamps up to year 294,247 and returns a result. This happens for TRY casts too, so try_cast raises instead of returning a value.

Query Comet
CAST(ts AS DATE) CometNativeException: Cast error: Cannot convert TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports dates between Jan 1, 262145 BCE and Dec 31, 262143 CE
CAST(ts AS STRING) same error
try_cast(ts AS DATE) same error
CAST(dt AS TIMESTAMP) native panic at spark-expr/src/conversion_funcs/temporal.rs:74:34
Steps to reproduce

As a SQL file test (spark/src/test/resources/sql-tests/...), default Spark 4.1 profile:

statement
CREATE TABLE tsr(ts timestamp, dt date) USING parquet

statement
INSERT INTO tsr VALUES
  (timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
  (timestamp_micros(0), date_from_unix_date(0))

query
SELECT CAST(ts AS DATE) FROM tsr

query
SELECT CAST(ts AS STRING) FROM tsr

query
SELECT try_cast(ts AS DATE) FROM tsr

query
SELECT CAST(dt AS TIMESTAMP) FROM tsr

timestamp_micros(9000000000000000000) is in year 287,000 or so, and date_from_unix_date(100000000) in year 275,000 or so; both are valid Spark values.

Expected behavior

The same results as Spark. Where the native kernel cannot represent the value, the serde should fall back to Spark rather than fail or panic.

Additional context

The error paths:

  • pre_timestamp_cast in native/spark-expr/src/utils.rs (as_datetime(...).ok_or_else(datetime_cast_err)), used by timestamp to string and timestamp to date for every eval mode.
  • timestamp_ntz_to_timestamp and cast_timestamp_to_ntz in the same file, same pattern.
  • Date to timestamp in native/spark-expr/src/conversion_funcs/temporal.rs: epoch + chrono::Duration::days(d) panics. The comment above it says chrono's range covers every i32 day count, but it does not.
  • test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs asserts the error for i64::MAX.

Found while auditing which expressions can safely run as TopK later keys in https://github.com/apache/datafusion-comet/pull/7457 , which now treats every timestamp cast as able to fail. Suggested triage: priority:high (native panic), area:expressions.

Lingua principale
Scala
Stelle
1.3k
Fork
414
Merge medio
3g 5h
PR unite (30g)
210

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.