Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

No extension type registry — consumers must manually decode FixedSizeBinary

Open
#423 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
typescript
Domain
api, data

Research direction

Start by locating the get.mjs path and visitor logic mentioned in the issue, then trace how vector.get(i) reads FixedSizeBinary values and field metadata. Compare that path with the proposed registerExtensionType API and determine how matching and custom getters would fit. Done means registered extension types can replace raw values for matching columns, with write-path serialization treated as an additional requirement.

Written by the indexing model from the issue text.

Description

Summary

Arrow JS has no mechanism to register custom getters for Arrow extension types. Columns with ARROW:extension:name and ARROW:extension:metadata field metadata always return raw bytes from get(). Every consumer must independently check metadata and decode values.

Background

The Arrow Extension Type spec (format docs) allows producers to annotate fields with semantic type information via metadata:

  • ARROW:extension:name — type identifier (e.g., "arrow.uuid", "arrow.opaque")
  • ARROW:extension:metadata — serialized type parameters (e.g., {"type_name": "hugeint", "vendor_name": "DuckDB"})

Other Arrow implementations provide extension type registration:

  • Arrow C++: arrow::ExtensionType — register a subclass with RegisterExtensionType(), and IPC deserialization automatically produces typed arrays with custom accessors
  • Arrow Python: pyarrow.ExtensionType — register with register_extension_type(), custom __arrow_ext_deserialize__ decodes IPC data into Python objects
  • Arrow Rust: arrow::datatypes::ExtensionType trait

Arrow JS has no equivalent. Extension types are preserved in field metadata but get() returns the raw storage value (e.g., Uint8Array for FixedSizeBinary).

Impact

DuckDB with arrow_lossless_conversion=true serializes several types as Arrow extension types:

DuckDB Type Arrow Storage Extension Name Bytes
HUGEINT FixedSizeBinary[16] arrow.opaque 16-byte two's complement signed int
UHUGEINT FixedSizeBinary[16] arrow.opaque 16-byte unsigned int
TIME WITH TIME ZONE FixedSizeBinary[8] arrow.opaque packed micros + offset
UUID FixedSizeBinary[16] arrow.uuid 16 raw bytes
BIGNUM Binary arrow.opaque 3-byte header + big-endian magnitude
VARINT Binary arrow.opaque same as BIGNUM
BIT Binary arrow.opaque padding byte + bit data

For each of these, consumers must:

  1. Check field.metadata.get("ARROW:extension:metadata")
  2. Parse the JSON to get type_name
  3. Read raw bytes from column.data[0].values at the correct offset
  4. Interpret the binary encoding (two's complement, packed bitfields, etc.)

This is ~100 lines of manual decoding in our codebase, repeated by every consumer that reads DuckDB Arrow output.

Proposal

Add an extension type registry, similar to C++/Python:

import { registerExtensionType } from 'apache-arrow';

registerExtensionType({
  name: 'arrow.opaque',         // matches ARROW:extension:name
  match: (metadata) => {        // optional: filter by extension metadata
    const parsed = JSON.parse(metadata);
    return parsed.type_name === 'hugeint';
  },
  get: (data, index) => {       // custom getter, replaces default
    const dv = new DataView(data.values.buffer, data.values.byteOffset + index * 16, 16);
    const lo = dv.getBigUint64(0, true);
    const hi = dv.getBigUint64(8, true);
    const raw = lo | (hi << 64n);
    if (raw & (1n << 127n)) {
      const mask = (1n << 128n) - 1n;
      return -(((raw ^ mask) + 1n) & mask);
    }
    return raw;
  },
});

After registration, vector.get(i) on a HUGEINT column would return a BigInt directly instead of a Uint8Array.

This could also support a serialize method for the write path, making round-trip extension types fully supported.

Alternatives

  • Do nothing: consumers continue to manually decode. Works, but fragile and duplicated.
  • Vendor-specific packages: e.g., @duckdb/arrow-extensions that monkey-patches Arrow's visitor. Feasible but hacky.
  • Local fork of get.mjs: what we currently do via Vite alias. Maintenance burden.

Context

We maintain a DuckDB WASM frontend that displays query results through Arrow IPC. Every DuckDB extension type requires custom byte-level decoding because Arrow JS can't be taught about them. The same decoding logic would need to be written by anyone consuming DuckDB, Spark, or other engines that use Arrow extension types in JS.

Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
17h 55m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow-js

All issues in apache/arrow-js

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.