`Data.valueOffsets` is longer than `length + 1` after an IPC round trip

Start with addBuffer in src/visitor/vectorassembler.ts and the corresponding region handling in src/visitor/vectorloader.ts, then compare both with _sliceBuffers in data.ts. Run the supplied 4-, 5-, and 6-row IPC repro, including the List case. Done means the intended logical offset length is established, documented or tested, and padding does not cause reconstructed data to report an extra row.

Describe the bug, including details regarding any error messages, version, and platform.

We hit an issue downstream where picking returned the wrong row, but only for batches with an even number of rows. I traced it to our code iterating over Data.valueOffsets. After an IPC round trip, the array includes an extra trailing zero for these batches.

Is valueOffsets intended to expose the full backing buffer, including padding, or only the logical offsets? Data.slice() trims it to length + 1, which made the difference surprising.

Repro

Reproduced on: apache-arrow 21.1.0, Node v22.20.0

import { Table, Utf8, tableFromIPC, tableToIPC, vectorFromArray } from "apache-arrow";

function check(n) {
  const values = Array.from({ length: n }, (_, i) => `v${i}`);
  const table = new Table({ s: vectorFromArray(values, new Utf8()) });
  const after = tableFromIPC(tableToIPC(table)).getChild("s").data[0];
  console.log({
    rows: n,
    expected: n + 1,
    actual: after.valueOffsets.length,
    tail: Array.from(after.valueOffsets.slice(-3)),
    afterSlice: after.slice(0, n).valueOffsets.length,
  });
}

check(4);
check(5);
check(6);

Gives:

{ rows: 4, expected: 5, actual: 6, tail: [ 6, 8, 0 ], afterSlice: 5 }
{ rows: 5, expected: 6, actual: 6, tail: [ 6, 8, 10 ], afterSlice: 6 }
{ rows: 6, expected: 7, actual: 8, tail: [ 10, 12, 0 ], afterSlice: 7 }

List behaves the same. Using [[0, 1], [2, 3], [4, 5], [6, 7]] as the values and new List(new Field("item", new Int32(), true)) as the type produces valueOffsets of [0, 2, 4, 6, 8, 0] after the IPC round trip.

An even row count needs an odd number of 32 bit offsets, which is 4 bytes short of an 8 byte boundary, so one zero offset gets appended. Odd row counts come out at exactly length + 1.

Traces

addBuffer in visitor/vectorassembler.ts rounds the declared region length up to 8 bytes:

const byteLength = (values.byteLength + 7) & ~7;

so the padding sits inside the BufferRegion the writer declares, and the loader hands the whole region back as the typed array. The format allows this. The Buffer.length docs in format/Schema.fbs say padding bytes "do not need to be accounted for in the size here", so a reader cannot assume the declared length excludes padding.

Observations

Data.slice() already clamps. _sliceBuffers in data.ts does arr.subarray(offset, offset + length + 1) for the offsets buffer, which explains the afterSlice behaviour above. I couldn’t find documentation explaining whether callers should expect this difference between loaded and sliced data.

Suggested fix

Would it make sense to trim the offsets view to length + 1 when loading IPC data? That would make it consistent with Data.slice() without copying the buffer. I’m unsure whether this belongs in the loader or whether exposing the full buffer is intentional.

When length is omitted, makeData defaults to valueOffsets.length - 1 for Utf8, LargeUtf8, Binary and LargeBinary, and List. Reconstructing the loaded Utf8 data from its buffers without explicitly passing length therefore reports one extra row for the even-row examples above.

Builders also over-allocate: vectorFromArray with four Utf8 rows produces valueOffsets.length === 16 before any IPC. I’m unsure whether trimming should be specific to IPC loading or apply more broadly to these variable-length types.