Native timestamp casts fail or panic for years beyond chrono's range

未关闭
  • 哪里不对？
#7,471 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
56/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
rust, scala, spark
领域
backend, databases

调研方向

Start with pre_timestamp_cast and the timestamp conversion functions in native/spark-expr/src/utils.rs, then inspect date-to-timestamp handling in native/spark-expr/src/conversion_funcs/temporal.rs and test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs. Reproduce with the SQL cases in spark/src/test/resources/sql-tests/... under the default Spark 4.1 profile. Done means out-of-range values return Spark-compatible results through fallback, including TRY casts, without native errors or panics.

由索引模型根据 Issue 内容生成。

描述

requires-triage
Describe the bug

Native casts to and from timestamps fail for values whose year is outside chrono's range (about ±262,143), while Spark supports timestamps up to year 294,247 and returns a result. This happens for TRY casts too, so try_cast raises instead of returning a value.

Query Comet
CAST(ts AS DATE) CometNativeException: Cast error: Cannot convert TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports dates between Jan 1, 262145 BCE and Dec 31, 262143 CE
CAST(ts AS STRING) same error
try_cast(ts AS DATE) same error
CAST(dt AS TIMESTAMP) native panic at spark-expr/src/conversion_funcs/temporal.rs:74:34
Steps to reproduce

As a SQL file test (spark/src/test/resources/sql-tests/...), default Spark 4.1 profile:

statement
CREATE TABLE tsr(ts timestamp, dt date) USING parquet

statement
INSERT INTO tsr VALUES
  (timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
  (timestamp_micros(0), date_from_unix_date(0))

query
SELECT CAST(ts AS DATE) FROM tsr

query
SELECT CAST(ts AS STRING) FROM tsr

query
SELECT try_cast(ts AS DATE) FROM tsr

query
SELECT CAST(dt AS TIMESTAMP) FROM tsr

timestamp_micros(9000000000000000000) is in year 287,000 or so, and date_from_unix_date(100000000) in year 275,000 or so; both are valid Spark values.

Expected behavior

The same results as Spark. Where the native kernel cannot represent the value, the serde should fall back to Spark rather than fail or panic.

Additional context

The error paths:

  • pre_timestamp_cast in native/spark-expr/src/utils.rs (as_datetime(...).ok_or_else(datetime_cast_err)), used by timestamp to string and timestamp to date for every eval mode.
  • timestamp_ntz_to_timestamp and cast_timestamp_to_ntz in the same file, same pattern.
  • Date to timestamp in native/spark-expr/src/conversion_funcs/temporal.rs: epoch + chrono::Duration::days(d) panics. The comment above it says chrono's range covers every i32 day count, but it does not.
  • test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs asserts the error for i64::MAX.

Found while auditing which expressions can safely run as TopK later keys in https://github.com/apache/datafusion-comet/pull/7457 , which now treats every timestamp cast as able to fail. Suggested triage: priority:high (native panic), area:expressions.

主要语言
Scala
星标
1.3k
派生
414
平均合并
3 天 4 小时
30 天内合并 PR
202

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion-comet 的其他 Issue

查看 apache/datafusion-comet 的全部 Issue

相似的 Issue

更多 Scala Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。