Native timestamp casts fail or panic for years beyond chrono's range

Abierto
  • ¿Algo no está bien?
#7,471 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
56/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust, scala, spark
Área
backend, databases

Línea de trabajo

Start with pre_timestamp_cast and the timestamp conversion functions in native/spark-expr/src/utils.rs, then inspect date-to-timestamp handling in native/spark-expr/src/conversion_funcs/temporal.rs and test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs. Reproduce with the SQL cases in spark/src/test/resources/sql-tests/... under the default Spark 4.1 profile. Done means out-of-range values return Spark-compatible results through fallback, including TRY casts, without native errors or panics.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

requires-triage
Describe the bug

Native casts to and from timestamps fail for values whose year is outside chrono's range (about ±262,143), while Spark supports timestamps up to year 294,247 and returns a result. This happens for TRY casts too, so try_cast raises instead of returning a value.

Query Comet
CAST(ts AS DATE) CometNativeException: Cast error: Cannot convert TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports dates between Jan 1, 262145 BCE and Dec 31, 262143 CE
CAST(ts AS STRING) same error
try_cast(ts AS DATE) same error
CAST(dt AS TIMESTAMP) native panic at spark-expr/src/conversion_funcs/temporal.rs:74:34
Steps to reproduce

As a SQL file test (spark/src/test/resources/sql-tests/...), default Spark 4.1 profile:

statement
CREATE TABLE tsr(ts timestamp, dt date) USING parquet

statement
INSERT INTO tsr VALUES
  (timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
  (timestamp_micros(0), date_from_unix_date(0))

query
SELECT CAST(ts AS DATE) FROM tsr

query
SELECT CAST(ts AS STRING) FROM tsr

query
SELECT try_cast(ts AS DATE) FROM tsr

query
SELECT CAST(dt AS TIMESTAMP) FROM tsr

timestamp_micros(9000000000000000000) is in year 287,000 or so, and date_from_unix_date(100000000) in year 275,000 or so; both are valid Spark values.

Expected behavior

The same results as Spark. Where the native kernel cannot represent the value, the serde should fall back to Spark rather than fail or panic.

Additional context

The error paths:

  • pre_timestamp_cast in native/spark-expr/src/utils.rs (as_datetime(...).ok_or_else(datetime_cast_err)), used by timestamp to string and timestamp to date for every eval mode.
  • timestamp_ntz_to_timestamp and cast_timestamp_to_ntz in the same file, same pattern.
  • Date to timestamp in native/spark-expr/src/conversion_funcs/temporal.rs: epoch + chrono::Duration::days(d) panics. The comment above it says chrono's range covers every i32 day count, but it does not.
  • test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs asserts the error for i64::MAX.

Found while auditing which expressions can safely run as TopK later keys in https://github.com/apache/datafusion-comet/pull/7457 , which now treats every timestamp cast as able to fail. Suggested triage: priority:high (native panic), area:expressions.

Lenguaje dominante
Scala
Estrellas
1.3k
Forks
414
Merge medio
3 d 5 h
PR fusionados (30 d)
210

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.