Hacktoberfest 2026:维护者为十月标记出来的 issue,仍然开放、适合新手。 浏览 Hacktoberfest issue

datalake_fdw: timestamp columns in units other than microseconds

未关闭
#1,990 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

@MisterRaindrop 已经在做这个了。

开始于 2026年9月13日。

评估

这个 Issue 还没有评估数据。

描述

datalake
Summary

The Parquet reader in contrib/datalake_fdw (#1951) reads only microsecond timestamps (tsu:), which is what Iceberg defines and what this module writes. Files written by other systems carry other units, and today every one of them is refused with a column stored as Arrow type "tsm:..." cannot be read as timestamp.

Cases
  • Millisecond columns (TIMESTAMP_MILLIS): common in Parquet written by Spark with spark.sql.parquet.outputTimestampType=TIMESTAMP_MILLIS, by Hive, and by many ETL tools. Multiplying by 1000 loses nothing. The question is whether a lake table should read a file whose type is not the table's type; Iceberg's spec says data files carry the table's types, so accepting them is a lenience, not a requirement.
  • Nanosecond columns (TIMESTAMP_NANOS, Iceberg v3 timestamp_ns): dividing by 1000 truncates. Refuse, or truncate and say so.
  • INT96 is already handled by coercing to microseconds. One caveat, Arrow's rather than ours (reproduced with pyarrow 21 and the same setting): Arrow's microsecond conversion assumes the nanos-of-day half is non-negative, which Spark/Hive/Impala guarantee; pyarrow's deprecated INT96 writer stores a negative one for instants before 1970 and those read wrong. Coercing to nanoseconds instead would fix that one case and break every date outside 1677..2262, including the 9999-12-31 sentinels warehouses keep. Worth an upstream report.
Where

format/arrow_decode.c: dl_arrow_decode_check() decides what a timestamp column accepts, dl_arrow_decode_value() converts. The TIMESTAMP/TIMESTAMPTZ case already distinguishes zoned from unzoned by whether the format string names a zone.

Deferred from #1951 on purpose.

主要语言
C
星标
1.4k
派生
248
平均合并
4 天 10 小时
30 天内合并 PR
40

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

apache/cloudberry 的其他 Issue

查看 apache/cloudberry 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。