Document that `date_bin` with an explicit origin drifts an hour across a DST transition
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 74/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- databases, documentation
Research direction
Start at the date_bin documentation entry point and reproduce the supplied DataFusion 55.0.0 query across the America/Denver DST transition. Document that an explicit origin fixes an instant and can drift from local midnight, then point readers to date_trunc for calendar units and the stated to_local_time pattern for non-calendar strides.
Written by the indexing model from the issue text.
Description
Describe the bug
Passing date_bin an explicit origin chosen to align bins to local midnight works until the timezone's offset changes, then silently shifts by an hour. There is no error and no warning.
This is not a DataFusion bug. PostgreSQL 15.19 and DuckDB 1.5.2 return identical results (see below). All three engines bin on the instant from a fixed origin, so the drift is inherent. What is missing is documentation: the pattern looks correct in every test that does not cross a DST transition.
CREATE OR REPLACE VIEW v AS SELECT arrow_cast(c, 'Timestamp(Second, Some("America/Denver"))') AS t
FROM (VALUES (TIMESTAMP '2024-03-09 06:30:00'), (TIMESTAMP '2024-03-11 06:30:00')) s(c);
SELECT t,
date_bin(INTERVAL '1 day', t) AS no_origin,
date_bin(INTERVAL '1 day', t,
arrow_cast(TIMESTAMP '2024-03-01 07:00:00','Timestamp(Second, Some("UTC"))')) AS with_origin
FROM v;
+---------------------------+---------------------------+---------------------------+
| t | no_origin | with_origin |
+---------------------------+---------------------------+---------------------------+
| 2024-03-09T06:30:00-07:00 | 2024-03-08T17:00:00-07:00 | 2024-03-09T00:00:00-07:00 |
| 2024-03-11T06:30:00-06:00 | 2024-03-10T18:00:00-06:00 | 2024-03-11T01:00:00-06:00 |
+---------------------------+---------------------------+---------------------------+
The origin 2024-03-01 07:00:00Z is 2024-03-01 00:00 MST, so bins land on local midnight — until America/Denver springs forward on 2024-03-10. After that the same expression returns 01:00:00-06:00.
To Reproduce
The query above, on DataFusion 55.0.0 (da89c7c85b).
Other engines
Measured with TimeZone = 'America/Denver' for the same instants and origin:
-- PostgreSQL 15.19
SELECT t, date_bin('1 day', t, '2024-03-01 07:00:00+00'::timestamptz), date_trunc('day', t)
FROM (VALUES ('2024-03-09 13:30:00+00'::timestamptz), ('2024-03-11 12:30:00+00'::timestamptz)) v(t);
-- DuckDB 1.5.2 has no date_bin; time_bucket with an origin is the equivalent
SELECT t, time_bucket(INTERVAL '1 day', t, TIMESTAMPTZ '2024-03-01 07:00:00+00'), date_trunc('day', t)
FROM (VALUES (TIMESTAMPTZ '2024-03-09 13:30:00+00'), (TIMESTAMPTZ '2024-03-11 12:30:00+00')) v(t);
t |
with origin | date_trunc('day', t) |
|---|---|---|
2024-03-09 06:30:00-07 |
2024-03-09 00:00:00-07 |
2024-03-09 00:00:00-07 |
2024-03-11 06:30:00-06 |
2024-03-11 01:00:00-06 |
2024-03-11 00:00:00-06 |
Both match DataFusion's output exactly.
Expected behavior
Document it on date_bin: an origin fixes an instant, so it does not keep bins on local midnight across a DST transition. Point users at date_trunc for calendar units, and at date_bin(<stride>, to_local_time(t AT TIME ZONE '<zone>')) for strides that are not calendar units. Changing the behaviour would move DataFusion away from PostgreSQL and DuckDB.
Related: #10602 (local-calendar binning) and #25167 (date_bin and date_trunc disagree on timezone-aware input).
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 354
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/datafusion#25266 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion#25213 · 1 comment ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
apache/datafusion#24998 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion#24913 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/datafusion#24911 ·
All issues in apache/datafusion
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100