feat(writer): write vortex.union (declared in Rust's unstable2026.10.0 edition)

Aperta
  • Qualcosa non va?
#660 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, rust
Ambito
backend, databases

Direzione di ricerca

Read #442 for the union wire layout, then inspect UnionEncodingEncoder registration in writer.encode and the cascade logic in vortex-compressor/src/compressor/cascade.rs. Check edition-family handling against ADR 0023 and the unstable declarations in vortex/editions/unstable/unstable2026.10.0.toml. Done means writing and cascading unions with unstable-edition gating, correcting docs/compatibility.md, and adding cross-language coverage; the issue lists a Rust-native harness as the first testing option.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Write side of #442. #442 deferred the writer because vortex.union was in no Rust edition ("Revisit when an edition declares it"). That is no longer true: Rust's unstable2026.10.0 edition (vortex/editions/unstable/unstable2026.10.0.toml, vortex-edition/src/declarations/unstable/mod.rs) lists vortex.union. docs/compatibility.md (lines 39, 150) still says "in no edition" and needs correcting as part of this.

Scope

  • UnionEncodingEncoder in writer.encode, registered in WriteRegistry.Builder#registerDefaults(): type ids (U8, nullable iff the union is) + one row-aligned child per variant, no metadata, no buffers (Rust rejects either). Layout per #442.
  • Cascading: compress type_ids and each child independently and rebuild the union, as vortex-compressor/src/compressor/cascade.rs does.
  • Edition gating: Rust now has an unstable family; EditionFamily is CORE/PREVIEW/ZSTD. Decide whether to add it (ADR 0023 says closed enum mirroring Rust's families, so probably yes) and make a default write never emit vortex.union.
  • Update docs/compatibility.md, AGENTS.md if the edition family changes, CHANGELOG.

Testing (vortex-jni cannot help: vortex-arrow bails on DataType::Union)

Ideas, cheapest first:

  1. Rust-native cargo harness (as for the Pco encoder): a small Rust program linking vortex-array/vortex-file that (a) reads a Java-written file and dumps the array tree + values, and (b) builds a UnionArray and writes it with the unstable edition, so Java reads real Rust output. This also gives the #442 decoder its first real-producer coverage.
  2. Rust-side vtx/vortex-tui browse on a Java-written file, as a quick manual smoke check.
  3. Byte-compare node trees: write the same logical union through the Rust harness and Java, diff the ArrayNode trees (the replicate-JNI-on-size-gaps approach).
  4. Reuse Rust's own test vectors in vortex-array/src/arrays/union/tests as unit fixtures.
  5. Keep the hand-built reader tests from #442; add random unions via a seeded @MethodSource (variant counts 1..256, non-consecutive type ids, null tags, nested union children).

Open question: is the unstable edition's vortex.union wire layout final? It makes no stability promise, so changes may follow.

Lingua principale
Java
Stelle
3
Fork
2
Merge medio
9h 55m
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dfa1/vortex-java

Tutte le issue di dfa1/vortex-java

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.