feat(writer): write vortex.union (declared in Rust's unstable2026.10.0 edition)

Open
  • Is something wrong?
#660 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java, rust
Domain
backend, databases

Research direction

Read #442 for the union wire layout, then inspect UnionEncodingEncoder registration in writer.encode and the cascade logic in vortex-compressor/src/compressor/cascade.rs. Check edition-family handling against ADR 0023 and the unstable declarations in vortex/editions/unstable/unstable2026.10.0.toml. Done means writing and cascading unions with unstable-edition gating, correcting docs/compatibility.md, and adding cross-language coverage; the issue lists a Rust-native harness as the first testing option.

Written by the indexing model from the issue text.

Description

enhancement

Write side of #442. #442 deferred the writer because vortex.union was in no Rust edition ("Revisit when an edition declares it"). That is no longer true: Rust's unstable2026.10.0 edition (vortex/editions/unstable/unstable2026.10.0.toml, vortex-edition/src/declarations/unstable/mod.rs) lists vortex.union. docs/compatibility.md (lines 39, 150) still says "in no edition" and needs correcting as part of this.

Scope

  • UnionEncodingEncoder in writer.encode, registered in WriteRegistry.Builder#registerDefaults(): type ids (U8, nullable iff the union is) + one row-aligned child per variant, no metadata, no buffers (Rust rejects either). Layout per #442.
  • Cascading: compress type_ids and each child independently and rebuild the union, as vortex-compressor/src/compressor/cascade.rs does.
  • Edition gating: Rust now has an unstable family; EditionFamily is CORE/PREVIEW/ZSTD. Decide whether to add it (ADR 0023 says closed enum mirroring Rust's families, so probably yes) and make a default write never emit vortex.union.
  • Update docs/compatibility.md, AGENTS.md if the edition family changes, CHANGELOG.

Testing (vortex-jni cannot help: vortex-arrow bails on DataType::Union)

Ideas, cheapest first:

  1. Rust-native cargo harness (as for the Pco encoder): a small Rust program linking vortex-array/vortex-file that (a) reads a Java-written file and dumps the array tree + values, and (b) builds a UnionArray and writes it with the unstable edition, so Java reads real Rust output. This also gives the #442 decoder its first real-producer coverage.
  2. Rust-side vtx/vortex-tui browse on a Java-written file, as a quick manual smoke check.
  3. Byte-compare node trees: write the same logical union through the Rust harness and Java, diff the ArrayNode trees (the replicate-JNI-on-size-gaps approach).
  4. Reuse Rust's own test vectors in vortex-array/src/arrays/union/tests as unit fixtures.
  5. Keep the hand-built reader tests from #442; add random unions via a seeded @MethodSource (variant counts 1..256, non-consecutive type ids, null tags, nested union children).

Open question: is the unstable edition's vortex.union wire layout final? It makes no stability promise, so changes may follow.

Dominant language
Java
Stars
3
Forks
2
Avg merge
9h 19m
Merged PRs (30d)
154

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dfa1/vortex-java

All issues in dfa1/vortex-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.