[Vector] Sparse UnionVector re-serialization rewrites schema type ids to MinorType ordinals, breaking cross-implementation interop

The issue is in UnionVector's getField() method, which incorrectly uses MinorType ordinals for type ids. Start by examining the UnionVector class and its getField() implementation. Look at how type ids are derived during serialization. The failing test case is provided; run it to reproduce the error. Check the ArrowType.Union handling and the interaction with Types.MinorType. The fix should ensure wire type ids are preserved in the schema.

Summary

Reading a spec-canonical sparse union produced by another Arrow implementation and then re-serializing it through arrow-java rewrites the schema's union type ids from the wire values [0, 1] to the Types.MinorType ordinals [24, 3], while the types (type-id) buffer is left unchanged at [0, 1]. The result is a self-inconsistent union: the buffer selects type id 0, which no longer appears in the field's declared type ids {24, 3}.

Strict Arrow consumers reject the output. Arrow C++/pyarrow RecordBatch.validate(full=True) fails with:

Invalid: In column 0: Invalid: Union value at position 0 has invalid type id 0

(Some readers, e.g. DuckDB, tolerate it by treating the type-id buffer positionally, which masks the problem until a strict consumer sees it.)

Version and platform

• Reproduced on arrow-java 18.1.0 and 19.0.0 (OpenJDK 25.0.2, macOS arm64). Confirmed still present after upgrading our stack to 19.0.0 — the round-trip produces the same [24, 3] schema type ids over a [0, 1] type-id buffer.

(An earlier revision of this report briefly retracted the 19.0.0 claim while our build was still transitively pulling 18.1.0; a real 19.0.0 build has since confirmed it reproduces.)

Steps to reproduce (pure arrow-java, no third-party runtime)

The input is a canonical sparse union sparse_union<name: string=0, age: int16=1> with a 2-row type-id buffer [0, 1], produced by pyarrow and serialized to an Arrow IPC stream (base64 below). arrow-java reads it, transfers the vectors into a fresh root (a generic pass-through), and writes it back:

byte[] input = Base64.getDecoder().decode(CANONICAL_UNION_IPC_B64);
try (BufferAllocator alloc = new RootAllocator()) {
  VectorSchemaRoot outRoot;
  try (ArrowStreamReader reader = new ArrowStreamReader(new ByteArrayInputStream(input), alloc)) {
    reader.loadNextBatch();
    VectorSchemaRoot in = reader.getVectorSchemaRoot();
    ArrowType.Union inType = (ArrowType.Union) in.getSchema().getFields().get(0).getType();
    System.out.println("INPUT  Union.typeIds = " + Arrays.toString(inType.getTypeIds())); // [0, 1]

    List<FieldVector> moved = new ArrayList<>();
    for (FieldVector v : in.getFieldVectors()) {
      TransferPair tp = v.getTransferPair(alloc);
      tp.transfer();
      moved.add((FieldVector) tp.getTo());
    }
    outRoot = new VectorSchemaRoot(moved);
    outRoot.setRowCount(in.getRowCount());
  }
  ByteArrayOutputStream sink = new ByteArrayOutputStream();
  try (ArrowStreamWriter w = new ArrowStreamWriter(outRoot, null, Channels.newChannel(sink))) {
    w.start(); w.writeBatch(); w.end();
  }
  outRoot.close();
  try (ArrowStreamReader reader = new ArrowStreamReader(new ByteArrayInputStream(sink.toByteArray()), alloc)) {
    reader.loadNextBatch();
    Field f = reader.getVectorSchemaRoot().getSchema().getFields().get(0);
    ArrowType.Union outType = (ArrowType.Union) f.getType();
    System.out.println("OUTPUT Union.typeIds = " + Arrays.toString(outType.getTypeIds())); // [24, 3]
  }
}

Output:

INPUT  Union.typeIds = [0, 1]
OUTPUT Union.typeIds = [24, 3]     // Utf8=24, SmallInt=3 — Types.MinorType ordinals

The type-id buffer is still [0, 1], so type id 0 is no longer declared.

CANONICAL_UNION_IPC_B64 (a valid pyarrow-produced union IPC stream):

/////+AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAEAAAAyP///wQAAAABAAAABAAAAIj///8AAAEOGAAAACQAAAAEAAAAAgAAAHAAAAAoAAAAAQAAAHUAAAAIAAgAAAAEAAgAAAAEAAAAAgAAAAAAAAABAAAAzP///wAAAQIQAAAAHAAAAAQAAAAAAAAAAwAAAGFnZQAIAAwACAAHAAgAAAAAAAABEAAAABAAFAAIAAYABwAMAAAAEAAQAAAAAAABBRAAAAAcAAAABAAAAAAAAAAEAAAAbmFtZQAAAAAEAAQABAAAAP/////oAAAAFAAAAAAAAAAMABYABgAFAAgADAAMAAAAAAMEABgAAAAoAAAAAAAAAAAACgAYAAwABAAIAAoAAAB8AAAAEAAAAAIAAAAAAAAAAAAAAAYAAAAAAAAAAAAAAAIAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAADAAAAAAAAAAYAAAAAAAAAAYAAAAAAAAAIAAAAAAAAAAAAAAAAAAAACAAAAAAAAAABAAAAAAAAAAAAAAAAwAAAAIAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAUAAAAGAAAAAAAAAEZyYW5reAAACQAFAAAAAAD/////AAAAAA==

Regenerate the input (Arrow C++/pyarrow), which round-trips it correctly:

import pyarrow as pa, base64
children = [pa.array(["Frank", "x"], pa.utf8()), pa.array([9, 5], pa.int16())]
type_ids = pa.array([0, 1], pa.int8())
u = pa.UnionArray.from_sparse(type_ids, children, field_names=["name", "age"])
batch = pa.RecordBatch.from_arrays([u], names=["u"])
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, batch.schema) as w:
    w.write_batch(batch)
print(base64.b64encode(sink.getvalue().to_pybytes()).decode())
# reading arrow-java's OUTPUT back in pyarrow raises:
#   In column 0: Invalid: Union value at position 0 has invalid type id 0

Expected

The re-serialized union should preserve the declared type ids [0, 1] (matching the type-id buffer), i.e. UnionVector.getField() should report the union's actual wire type ids rather than deriving them from Types.MinorType.ordinal().

Notes

• The corruption is in the schema/getField() type-id derivation, not the data buffers — getField() returns type ids based on MinorType ordinals rather than the union's wire type ids.
• Constructing a UnionVector fresh via setType(...) and calling getField() returns Union(Sparse, []) (empty type ids), a related manifestation of the same root cause.
• Searched existing reports: ARROW-1692 (dense/sparse detection on read, fixed 1.0.0) and ARROW-6145 (field metadata preservation, fixed 0.15.0) are adjacent but distinct from this type-id derivation on serialize.

Reported by Rusty Conover (Query Farm — https://query.farm), found while running a cross-implementation Arrow conformance suite.