`null_if_overflow_precision` resets decimal precision and scale, changing valid values
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 88/100
Research direction
Start in arrow-array/src/array/primitive_array.rs at null_if_overflow_precision and inspect the unary_opt result construction. Add regression coverage in arrow-array/tests/decimal_overflow_mask.rs, including the decimal widths that share this implementation, then run cargo test -p arrow-array --test decimal_overflow_mask -- --nocapture. Done means masked arrays retain their input datatype and surviving values keep their numerical meaning.
Written by the indexing model from the issue text.
Description
Describe the bug
Decimal128Array::null_if_overflow_precision resets the output datatype to
Decimal128(38,10) instead of preserving the input precision and scale. It
retains the unscaled integers, so this changes the numerical meaning of values
that survive the mask.
No overflow is needed to trigger the bug: a valid one-element
Decimal128(2,1) array containing 9.9 becomes a Decimal128(38,10) array
containing 0.0000000099.
Reproduced on Apache Arrow Rust main at
8208506f8f9ec193c08023ac2477d211d1d86d20
(workspace version 60.0.0), checked on 2026-09-29.
To reproduce
In an arrow-rs checkout, put this in
arrow-array/tests/decimal_overflow_mask.rs:
use arrow_array::{Array, Decimal128Array};
#[test]
fn overflow_mask_preserves_decimal_type() {
let input = Decimal128Array::from(vec![99])
.with_precision_and_scale(2, 1)
.unwrap();
input.validate_decimal_precision(2).unwrap();
let output = input.null_if_overflow_precision(2);
println!("before: {:?}, {}", input.data_type(), input.value_as_string(0));
println!("after: {:?}, {}", output.data_type(), output.value_as_string(0));
assert_eq!(output.value(0), 99);
assert!(!output.is_null(0));
assert_eq!(output.data_type(), input.data_type()); // fails
}
Run from the repository root:
cargo test -p arrow-array --test decimal_overflow_mask -- --nocapture
Observed output:
before: Decimal128(2, 1), 9.9
after: Decimal128(38, 10), 0.0000000099
assertion `left == right` failed
left: Decimal128(38, 10)
right: Decimal128(2, 1)
Expected behavior
Mask values that exceed the supplied precision without changing the array's
datatype or the numerical meaning of surviving values. In this example the
output should still be Decimal128(2,1) containing 9.9.
Apparent cause
null_if_overflow_precision
calls self.unary_opt::<_, T>(...). That generic operation constructs its
result with
PrimitiveArray::new,
whose constructor initializes the datatype from T::DATA_TYPE. For
Decimal128Type, that is the default (38,10), not the input array's runtime
precision and scale. The masking method does not restore that runtime datatype.
A caller can compensate by applying .with_data_type(input.data_type().clone())
to the result. Preserving the datatype in the masking method itself would avoid
requiring every caller to do so. Regression coverage should include the other
decimal widths that share this implementation.
Additional context
This is separate from float/string-to-decimal conversion overflow: the input
above is already a valid decimal array, and no conversion or rounding occurs.
It was found while investigating Variant extraction, but this reproducer uses
only arrow-array.
AI assistance: OpenAI Codex assisted with investigation, reproducer execution,
and drafting this report.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 146
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-rs
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/arrow-rs#11225 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
parquet-variant-compute: shred_variant panics on an object with duplicate field namesPossibly taken @Abhisheklearn12 claimed this 13 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
development-process enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/arrow-rs#9976 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 3 days
-
state:triage-needed
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Automattic/harper#4503 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Maintainers usually reply within 2 days