[JS] Decimal-type conversions result in errors/inaccuracies in Arrow JS
@trxcllnt is already working on this.
Since Sep 28, 2023.
Assessment
This issue has not been assessed yet.
Description
Describe the bug, including details regarding any error messages, version, and platform.
Thanks for all the work on Arrow! Unfortunately, I've run into problems in the JS API around both encoding and decoding of Decimal-typed data.
Encoding:
arrow = require('apache-arrow')
arrow.vectorFromArray([1, 12, 34], new arrow.Decimal(3, 18))
Results in TypeError: d.subarray is not a function
(Incidentally it also seems that the Decimal type constructor uses a different argument order than pyarrow.)
Decoding:
Since encoding failed in JS, let's do it in pyarrow:
import pyarrow as pa
v = pa.array([1, 12, 34], type=pa.decimal128(18, 3))
batch = pa.record_batch([v], names=['d'])
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, batch.schema) as writer:
writer.write_batch(batch)
sink.getvalue().hex()
and then we can carry the resulting byte string over to JavaScript and try to decode it:
const hex = 'FFFFFFFF780000001000000000000A000C000600050008000A000000000104000C000000080008000000040008000000040000000100000014000000100014000800060007000C00000010001000000000000107100000001C0000000400000000000000010000006400000008000C0004000800080000001200000003000000FFFFFFFF8800000014000000000000000C0016000600050008000C000C0000000003040018000000300000000000000000000A0018000C00040008000A0000003C00000010000000030000000000000000000000020000000000000000000000000000000000000000000000000000003000000000000000000000000100000003000000000000000000000000000000E8030000000000000000000000000000E02E0000000000000000000000000000D0840000000000000000000000000000FFFFFFFF00000000';
const bytes = Uint8Array.from(hex.match(/.{1,2}/g).map(s => parseInt(s, 16)));
const vec = arrow.tableFromIPC(bytes).getChild('d');
vec.get(0) // [1000, 0, 0, 0] <- byte array looks correct
`${vec.get(0)}` // "1000" <- incorrect, should be 1; ignores scale
Number(vec.get(0)) // 2.2136092888451462e+23 <- incorrect, should be 1
+vec.get(0) // TypeError: can't convert BigInt to number <- ???
While the basic byte encoding looks OK, all of the built-in conversions appear to be faulty.
Related: apache/arrow-js#100, apache/arrow-js#81
cc: @domoritz
Component(s)
JavaScript
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 17h 55m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-js
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 50/100
Similar issues
-
blocklist removal
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
MetaMask/eth-phishing-detect#296544 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
pastelsky/bundlephobia#1122 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
category/development priority/P2 scope/file-operations scope/testing type/enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Enatega Customer and Rider app: Add-ons price is not visible to customer after order is placed. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100