Native Parquet write falls back on int96RebaseModeInWrite=LEGACY even when timestamps are not written as INT96

Open Beginner friendly
  • Is something wrong?
#7,463 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
62/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust, scala, spark
Domain
data-engineering

Research direction

The fallback lives in NativeWriteUtils.legacyDatetimeRebaseWriteReason; start there and read how it inspects int96RebaseModeInWrite before checking the output timestamp type. Add a condition so the INT96 check applies only when spark.sql.parquet.outputTimestampType is INT96, then add a regression test with TIMESTAMP_MICROS and int96RebaseModeInWrite=LEGACY asserting the write stays native. Done when that test passes and the paragraph in docs/source/user-guide/latest/parquet.md is updated.

Written by the indexing model from the issue text.

Description

area:writer enhancement requires-triage
What is the problem the feature request solves?

Found in review of #7053. The native Parquet writer fallback added there, NativeWriteUtils.legacyDatetimeRebaseWriteReason, treats spark.sql.parquet.int96RebaseModeInWrite=LEGACY as a reason to fall back whenever the output has a timestamp column. That setting only governs the INT96 encoding. When spark.sql.parquet.outputTimestampType is TIMESTAMP_MICROS or TIMESTAMP_MILLIS, Spark writes INT64 and uses datetimeRebaseModeInWrite instead, so the INT96 setting has no effect on the output.

Reproduction: a timestamp column, outputTimestampType=TIMESTAMP_MICROS, datetimeRebaseModeInWrite=CORRECTED. Changing only int96RebaseModeInWrite from CORRECTED to LEGACY makes the helper return a fallback reason instead of None. Spark writes the same INT64 TIMESTAMP(MICROS) either way (for 1500-01-01 12:34:56.123456 both files contain -14831724303876544).

The fallback is safe, but an unrelated legacy INT96 setting disables native write acceleration for a write the native writer handles correctly.

Describe the potential solution

Only apply the INT96 condition when spark.sql.parquet.outputTimestampType is INT96. The native writer always writes INT64 microseconds and ignores outputTimestampType when writing, so the check has to read the session conf.

Add a regression test with a non-INT96 timestamp output and int96RebaseModeInWrite=LEGACY that asserts the write stays native and matches Spark's value. Update the paragraph in docs/source/user-guide/latest/parquet.md that names int96RebaseModeInWrite=LEGACY as a fallback trigger.

Additional context

Reviewer comment: https://github.com/apache/datafusion-comet/pull/7053#discussion_r4458640616

Follow-on to #7053, which should merge first.

Dominant language
Scala
Stars
1.3k
Forks
405
Avg merge
3d 1h
Merged PRs (30d)
277

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-comet

All issues in apache/datafusion-comet

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.