Temporal transforms put pre-epoch timestamps at `.999999` into the previous unit
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
Research direction
Start at DateTimeUtil.convertMicros and DateTimeUtil.convertNanos, then trace how the year, month, day, and hour transforms use them. Add regression coverage for the exact pre-epoch boundary values described in the issue and verify the transforms remain monotonic and inclusive projection no longer prunes the matching row.
Written by the indexing model from the issue text.
Description
Apache Iceberg version
main (development)
Query engine
Other
Please describe the bug 🐞
DateTimeUtil.convertMicros and DateTimeUtil.convertNanos return the previous unit for a pre-epoch timestamp in the first second of a unit when its fraction is .999999 (.999999999 for nanos). This affects the year, month, day and hour transforms.
var day = Transforms.day().bind(Types.TimestampType.withoutZone());
long midnight = -86_400_000_000L; // 1969-12-31T00:00:00
day.apply(midnight + 999_998); // -1
day.apply(midnight + 999_999); // -2, expected -1
day.apply(midnight + 1_000_000); // -1
The spec defines day as days from 1970-01-01, so 1969-12-31T00:00:00.999999 should be -1. The transform also stops being monotonic, which inclusive projection relies on, so Java's own scan skips a matching row:
// row 1969-12-31T00:00:00.999999 is written to partition ts_day = -2
Projections.inclusive(spec).project(greaterThanOrEqual("ts", midnight + 500_000));
// -> ts_day >= -1, so the file holding the row is pruned
Likelihood: low. It needs a pre-epoch timestamp, in the first second of an hour/day/month/year, with a fraction of exactly .999999 (.999999999 for nanos). Second- or millisecond-precision data never hits it. For random microsecond-precision pre-epoch values it's roughly 1 in 10^11 rows for day and 1 in 4×10^9 for hour.
Cause: the negative branch adds 1 to the fraction but not to the seconds:
long epochSecond = Math.floorDiv(micros, MICROS_PER_SECOND);
long nanoAdjustment = Math.floorMod(micros + 1, MICROS_PER_SECOND) * 1000;
At .999999, micros + 1 crosses into the next second but epochSecond does not, so the timestamp lands almost a second earlier. In the first second of a unit, that crosses into the previous unit.
Fix: Math.floorDiv(micros + 1, MICROS_PER_SECOND), and the same in convertNanos with NANOS_PER_SECOND. I checked the micros version against floor semantics for year, month, day and hour across ~47k values, including every unit boundary from 1900 to 1971: the current code is wrong for 3,446 of them, the fix for none.
iceberg-rust returns the calendar day for these values, see the iceberg-rust day transform fix.
Related: #18115 (same class of bug, rounding toward zero instead of down for pre-epoch values, but in microsecond conversion rather than the transforms), #9714 (transform rounding for negative values isn't specified). The +1/-1 adjustment comes from #1981.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
- Dominant language
- Java
- Stars
- 9.3k
- Forks
- 3.6k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 154
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg
-
improvement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg#18351 · 1 comment ·
Maintainers usually reply within 1 day
-
Parquet: StringReader decodes every occurrence of a dictionary value into a new StringPossibly taken @Neuw84 claimed this 10 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg#18259 · 2 comments ·
Maintainers usually reply within 1 day
-
RCK tests fail to bind port 8181 when run alongside the kafka-connect integration stackPossibly taken @Kurtiscwright claimed this 12 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Core: SchemaParser rejects integral JSON numbers for floating-point defaultsPossibly taken @laserninja claimed this 20 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
API: Binary truncate overflows with a positioned buffer and large widthPossibly taken @laserninja claimed this 20 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
bancolombia/scaffold-clean-architecture#1002 ·
Maintainers usually reply within 1 day
-
CalendarEventAttendance/get returns eventAttendanceStatus while the doc says attendanceStatusPossibly taken @chibenwa claimed this today. Openbug claude
Difficulty 1/5 Under an hour Newbie friendliness 90/100
linagora/tmail-backend#2697 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/skywalking#14120 ·
Maintainers usually reply within 1 day
-
[BUG] Case-insensitive search suggestions miss items when the JVM default locale is TurkishPossibly taken @thswlsqls claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
HMCL-dev/HMCL#6943 · 1 comment ·
Maintainers usually reply within 1 day