datalake_fdw: timestamp columns in units other than microseconds
@MisterRaindrop 已经在做这个了。
开始于 2026年9月13日。
评估
这个 Issue 还没有评估数据。
描述
Summary
The Parquet reader in contrib/datalake_fdw (#1951) reads only microsecond timestamps (tsu:), which is what Iceberg defines and what this module writes. Files written by other systems carry other units, and today every one of them is refused with a column stored as Arrow type "tsm:..." cannot be read as timestamp.
Cases
- Millisecond columns (
TIMESTAMP_MILLIS): common in Parquet written by Spark withspark.sql.parquet.outputTimestampType=TIMESTAMP_MILLIS, by Hive, and by many ETL tools. Multiplying by 1000 loses nothing. The question is whether a lake table should read a file whose type is not the table's type; Iceberg's spec says data files carry the table's types, so accepting them is a lenience, not a requirement. - Nanosecond columns (
TIMESTAMP_NANOS, Iceberg v3timestamp_ns): dividing by 1000 truncates. Refuse, or truncate and say so. - INT96 is already handled by coercing to microseconds. One caveat, Arrow's rather than ours (reproduced with pyarrow 21 and the same setting): Arrow's microsecond conversion assumes the nanos-of-day half is non-negative, which Spark/Hive/Impala guarantee; pyarrow's deprecated INT96 writer stores a negative one for instants before 1970 and those read wrong. Coercing to nanoseconds instead would fix that one case and break every date outside 1677..2262, including the 9999-12-31 sentinels warehouses keep. Worth an upstream report.
Where
format/arrow_decode.c: dl_arrow_decode_check() decides what a timestamp column accepts, dl_arrow_decode_value() converts. The TIMESTAMP/TIMESTAMPTZ case already distinguishes zoned from unzoned by whether the format string names a zone.
Deferred from #1951 on purpose.
- 主要语言
- C
- 星标
- 1.4k
- 派生
- 248
- 平均合并
- 4 天 10 小时
- 30 天内合并 PR
- 40
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/cloudberry 的其他 Issue
-
type: Bug
难度 2/5 1-3 小时 新手友好度 76/100
apache/cloudberry#1885 · 2 个 reaction ·
-
难度 2/5 1-3 小时 新手友好度 86/100
apache/cloudberry#1825 ·
-
type: Bug
难度 3/5 1-2 天 新手友好度 65/100
apache/cloudberry#2048 · 1 个 reaction ·
-
type: Bug
难度 4/5 3-5 天 新手友好度 40/100
apache/cloudberry#2047 ·
-
type: Bug
难度 4/5 3-5 天 新手友好度 45/100
apache/cloudberry#2046 · 1 条评论 ·
查看 apache/cloudberry 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 75/100
bradcypert/plum#53 ·
-
Component: GLib
难度 2/5 1-3 小时 新手友好度 70/100
-
难度 2/5 1-3 小时 新手友好度 75/100
-
Status: Opened
难度 2/5 1-3 小时 新手友好度 70/100
-
难度 2/5 1-3 小时 新手友好度 75/100
nextbsd/nextbsd-userland#285 ·