No extension type registry — consumers must manually decode FixedSizeBinary
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- typescript
Research direction
Start by locating the get.mjs path and visitor logic mentioned in the issue, then trace how vector.get(i) reads FixedSizeBinary values and field metadata. Compare that path with the proposed registerExtensionType API and determine how matching and custom getters would fit. Done means registered extension types can replace raw values for matching columns, with write-path serialization treated as an additional requirement.
Written by the indexing model from the issue text.
Description
Summary
Arrow JS has no mechanism to register custom getters for Arrow extension types. Columns with ARROW:extension:name and ARROW:extension:metadata field metadata always return raw bytes from get(). Every consumer must independently check metadata and decode values.
Background
The Arrow Extension Type spec (format docs) allows producers to annotate fields with semantic type information via metadata:
ARROW:extension:name— type identifier (e.g.,"arrow.uuid","arrow.opaque")ARROW:extension:metadata— serialized type parameters (e.g.,{"type_name": "hugeint", "vendor_name": "DuckDB"})
Other Arrow implementations provide extension type registration:
- Arrow C++:
arrow::ExtensionType— register a subclass withRegisterExtensionType(), and IPC deserialization automatically produces typed arrays with custom accessors - Arrow Python:
pyarrow.ExtensionType— register withregister_extension_type(), custom__arrow_ext_deserialize__decodes IPC data into Python objects - Arrow Rust:
arrow::datatypes::ExtensionTypetrait
Arrow JS has no equivalent. Extension types are preserved in field metadata but get() returns the raw storage value (e.g., Uint8Array for FixedSizeBinary).
Impact
DuckDB with arrow_lossless_conversion=true serializes several types as Arrow extension types:
| DuckDB Type | Arrow Storage | Extension Name | Bytes |
|---|---|---|---|
HUGEINT |
FixedSizeBinary[16] | arrow.opaque | 16-byte two's complement signed int |
UHUGEINT |
FixedSizeBinary[16] | arrow.opaque | 16-byte unsigned int |
TIME WITH TIME ZONE |
FixedSizeBinary[8] | arrow.opaque | packed micros + offset |
UUID |
FixedSizeBinary[16] | arrow.uuid | 16 raw bytes |
BIGNUM |
Binary | arrow.opaque | 3-byte header + big-endian magnitude |
VARINT |
Binary | arrow.opaque | same as BIGNUM |
BIT |
Binary | arrow.opaque | padding byte + bit data |
For each of these, consumers must:
- Check
field.metadata.get("ARROW:extension:metadata") - Parse the JSON to get
type_name - Read raw bytes from
column.data[0].valuesat the correct offset - Interpret the binary encoding (two's complement, packed bitfields, etc.)
This is ~100 lines of manual decoding in our codebase, repeated by every consumer that reads DuckDB Arrow output.
Proposal
Add an extension type registry, similar to C++/Python:
import { registerExtensionType } from 'apache-arrow';
registerExtensionType({
name: 'arrow.opaque', // matches ARROW:extension:name
match: (metadata) => { // optional: filter by extension metadata
const parsed = JSON.parse(metadata);
return parsed.type_name === 'hugeint';
},
get: (data, index) => { // custom getter, replaces default
const dv = new DataView(data.values.buffer, data.values.byteOffset + index * 16, 16);
const lo = dv.getBigUint64(0, true);
const hi = dv.getBigUint64(8, true);
const raw = lo | (hi << 64n);
if (raw & (1n << 127n)) {
const mask = (1n << 128n) - 1n;
return -(((raw ^ mask) + 1n) & mask);
}
return raw;
},
});
After registration, vector.get(i) on a HUGEINT column would return a BigInt directly instead of a Uint8Array.
This could also support a serialize method for the write path, making round-trip extension types fully supported.
Alternatives
- Do nothing: consumers continue to manually decode. Works, but fragile and duplicated.
- Vendor-specific packages: e.g.,
@duckdb/arrow-extensionsthat monkey-patches Arrow's visitor. Feasible but hacky. - Local fork of get.mjs: what we currently do via Vite alias. Maintenance burden.
Context
We maintain a DuckDB WASM frontend that displays query results through Arrow IPC. Every DuckDB extension type requires custom byte-level decoding because Arrow JS can't be taught about them. The same decoding logic would need to be written by anyone consuming DuckDB, Spark, or other engines that use Arrow extension types in JS.
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 17h 55m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-js
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 50/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
bcgov/bc-wallet-mobile#4761 · 1 comment ·
-
external-issue to-triage
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
area-deployment area-integrations triage:bot-seen
Difficulty 2/5 Half a day Newbie friendliness 86/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
refactor
Difficulty 2/5 1-3 hours Newbie friendliness 84/100