Spark `round()` on floating-point types diverges from Spark HALF_UP semantics
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Domain
- data-engineering
Research direction
Start in datafusion/spark/src/function/math/round.rs, especially the round_float entry point and its existing doc comment describing BigDecimal and HALF_UP behavior. Reproduce the 1.255 and 1.005 SQL examples, then add regression coverage showing Spark-compatible results for floating-point inputs, including both FloatType and DoubleType.
Written by the indexing model from the issue text.
Description
Describe the bug
The Spark-compatible round() function gives different results from Apache Spark when the input is a floating-point type (FloatType/DoubleType) and the value's binary representation is slightly off from its decimal literal.
Spark's RoundBase rounds a double as BigDecimal(d).setScale(scale, HALF_UP), where BigDecimal(Double) is java.math.BigDecimal.valueOf(d) — i.e. it parses the shortest round-trip decimal string of the double (Double.toString). DataFusion's round_float instead does naive binary-float arithmetic, (value * 10^scale).round() / 10^scale, which rounds the already-imprecise binary value and diverges at the half-way point.
To Reproduce
SELECT round(1.255::double, 2::int);
-- Spark: 1.26
-- DataFusion: 1.25
SELECT round(1.005::double, 2::int);
-- Spark: 1.01
-- DataFusion: 1.0
The cause is that 1.255 and 1.005 are stored as binary doubles a hair below the decimal value (1.2549999999999999..., 1.00499999999999989...). Spark sees the shortest decimal string ("1.255", "1.005") and applies HALF_UP, so the tie rounds away from zero. DataFusion multiplies the raw binary value by 100, which stays below the half-way point, and rounds down.
Expected behavior
Match Spark: round via the shortest round-trip decimal representation with HALF_UP (ties away from zero), for both DoubleType and FloatType (Spark widens float to double first via f.toDouble).
Additional context
The existing doc comment on round_float already describes the intended BigDecimal / HALF_UP behaviour; the implementation simply doesn't match it. I have a fix and will open a PR referencing this issue.
datafusion/spark/src/function/math/round.rs
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 354
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/datafusion#25266 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion#25213 · 1 comment ·
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
apache/datafusion#25168 · 1 comment ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
apache/datafusion#24998 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion#24913 ·
All issues in apache/datafusion
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
bitcoindevkit/bdk-ffi#1125 ·