Spark `round()` on floating-point types diverges from Spark HALF_UP semantics

Open Beginner friendly
#22,812 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
rust, spark, sql

Research direction

Start in datafusion/spark/src/function/math/round.rs, especially the round_float entry point and its existing doc comment describing BigDecimal and HALF_UP behavior. Reproduce the 1.255 and 1.005 SQL examples, then add regression coverage showing Spark-compatible results for floating-point inputs, including both FloatType and DoubleType.

Written by the indexing model from the issue text.

Description

Describe the bug

The Spark-compatible round() function gives different results from Apache Spark when the input is a floating-point type (FloatType/DoubleType) and the value's binary representation is slightly off from its decimal literal.

Spark's RoundBase rounds a double as BigDecimal(d).setScale(scale, HALF_UP), where BigDecimal(Double) is java.math.BigDecimal.valueOf(d) — i.e. it parses the shortest round-trip decimal string of the double (Double.toString). DataFusion's round_float instead does naive binary-float arithmetic, (value * 10^scale).round() / 10^scale, which rounds the already-imprecise binary value and diverges at the half-way point.

To Reproduce
SELECT round(1.255::double, 2::int);
-- Spark:      1.26
-- DataFusion: 1.25

SELECT round(1.005::double, 2::int);
-- Spark:      1.01
-- DataFusion: 1.0

The cause is that 1.255 and 1.005 are stored as binary doubles a hair below the decimal value (1.2549999999999999..., 1.00499999999999989...). Spark sees the shortest decimal string ("1.255", "1.005") and applies HALF_UP, so the tie rounds away from zero. DataFusion multiplies the raw binary value by 100, which stays below the half-way point, and rounds down.

Expected behavior

Match Spark: round via the shortest round-trip decimal representation with HALF_UP (ties away from zero), for both DoubleType and FloatType (Spark widens float to double first via f.toDouble).

Additional context

The existing doc comment on round_float already describes the intended BigDecimal / HALF_UP behaviour; the implementation simply doesn't match it. I have a fix and will open a PR referencing this issue.

datafusion/spark/src/function/math/round.rs

Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 8h
Merged PRs (30d)
354

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion

All issues in apache/datafusion

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.