Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[JS] Decimal-type conversions result in errors/inaccuracies in Arrow JS

Open
#77 7 comments 3 reactions 1 assignee View on GitHub

@trxcllnt is already working on this.

Since Sep 28, 2023.

Assessment

This issue has not been assessed yet.

Description

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.

Thanks for all the work on Arrow! Unfortunately, I've run into problems in the JS API around both encoding and decoding of Decimal-typed data.

Encoding:

arrow = require('apache-arrow')
arrow.vectorFromArray([1, 12, 34], new arrow.Decimal(3, 18))

Results in TypeError: d.subarray is not a function

(Incidentally it also seems that the Decimal type constructor uses a different argument order than pyarrow.)

Decoding:

Since encoding failed in JS, let's do it in pyarrow:

import pyarrow as pa
v = pa.array([1, 12, 34], type=pa.decimal128(18, 3))
batch = pa.record_batch([v], names=['d'])

sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, batch.schema) as writer:
   writer.write_batch(batch)
sink.getvalue().hex()

and then we can carry the resulting byte string over to JavaScript and try to decode it:

const hex = 'FFFFFFFF780000001000000000000A000C000600050008000A000000000104000C000000080008000000040008000000040000000100000014000000100014000800060007000C00000010001000000000000107100000001C0000000400000000000000010000006400000008000C0004000800080000001200000003000000FFFFFFFF8800000014000000000000000C0016000600050008000C000C0000000003040018000000300000000000000000000A0018000C00040008000A0000003C00000010000000030000000000000000000000020000000000000000000000000000000000000000000000000000003000000000000000000000000100000003000000000000000000000000000000E8030000000000000000000000000000E02E0000000000000000000000000000D0840000000000000000000000000000FFFFFFFF00000000';
const bytes = Uint8Array.from(hex.match(/.{1,2}/g).map(s => parseInt(s, 16)));
const vec = arrow.tableFromIPC(bytes).getChild('d');

vec.get(0)         // [1000, 0, 0, 0] <- byte array looks correct
`${vec.get(0)}`    // "1000" <- incorrect, should be 1; ignores scale
Number(vec.get(0)) // 2.2136092888451462e+23 <- incorrect, should be 1
+vec.get(0)        // TypeError: can't convert BigInt to number <- ???

While the basic byte encoding looks OK, all of the built-in conversions appear to be faulty.

Related: apache/arrow-js#100, apache/arrow-js#81
cc: @domoritz

Component(s)

JavaScript

Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
17h 55m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow-js

All issues in apache/arrow-js

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.