New module `mapper`: read and write Java records as typed rows

Offen
  • Stimmt etwas nicht?
#528 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@dfa1 arbeitet bereits daran.

Seit 17.10.2026.

  • #549 von @dfa1 — offen

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Feature
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
java
Bereich
database

Rechercherichtung

Start by reading the existing reader and writer modules, especially VortexReader, VortexWriter, and the ScanOptions and WriteOptions APIs; the issue names no specific file paths or tests. Define a new mapper module that depends on both modules and inspect how their column types and batches are represented. Done means Java records map to the listed basic types, projection reads validate names and types, annotations follow the described rules, and unsupported types fail clearly.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

enhancement help wanted PARKED

A small module, mapper, that maps a Java record to a Vortex struct schema, for both the reader and the writer. Same idea as the typed-record module we did for typesafe-java; inspired by Vorticity v0.1.0, whose [VortexRecord] declares the columns of a file.

Minimum requirements

  • A Java record is the schema: its components are the columns, in order, under their names (ColumnName).
  • Write: records in, a .vortex file out, through the existing VortexWriter. Taking a List<Record> is fine and convenient: the usual source is row oriented, so the writer pivots rows into column batches.
  • Read: a .vortex file in, records out, through the existing VortexReader.
  • A new mapper module that depends on reader and writer; neither of them depends on it (the writer never depends on the reader today, and this module is the only place that sees both).
  • An unsupported component type fails fast with a clear error.

Projection is the reading model

A columnar file is read by column, never select *. The read side must make that the obvious path and say so in the docs:

  • A record passed to read is a projection: only its components are decoded, so a record with 1 or 2 components reads 1 or 2 columns of a file that has many more (it maps to ScanOptions columns).
  • The record need not match the file's full schema: any subset of the file's columns, by name, is valid. A component with no matching column, or with a different type, fails fast.
  • Document it with an example: a wide file written from a full Trade record, read back through a small PriceOnly(double price) record. Do not show reading the whole schema as the default.

First round: basic types only

Mapped from the Rust DType variants, only the basic ones:

DType Java component
Bool boolean / Boolean
Primitive byte, short, int, long, float, double (boxed = nullable); unsigned via @Unsigned (see below); F16 is deferred
Utf8 String
Binary byte[]

Nullability follows the component type: a boxed type or a reference type is nullable, a Java primitive is not.

F16 is deferred until Valhalla. A short of raw half bits needs an annotation to tell it from I16, and that API would be replaced once the JDK has a half-float value type. Until then a record component cannot map to an F16 column: it fails fast with a clear error, and the array API still reads and writes F16 as short[].

Later rounds, not in this ticket: Decimal, List, FixedSizeList, Map, nested Struct (a record inside a record), Union, Variant, and Extension (dates, times, timestamps, UUID), plus Null.

Exploratory: annotations

The module is exploratory, so these are proposals to try, not commitments. Principle: a schema annotation (changes the DType) must agree on the read and write side and is checked fail-fast; a write-time annotation is ignored when reading.

  • Unsigned (schema): Java has no unsigned types, and the existing array API already stores U8..U64 in the same byte[]/short[]/int[]/long[] as I8..I64. Try @Unsigned on a same-width Java primitive (@Unsigned int is U32), bit-identical with no conversion; the user reads it with Integer.toUnsignedLong and friends. A lossless widening on read (U32 as long) is a possible later addition.
  • Forcing encodings (write-time): @Encoding("vortex.zstd", ...) on a component maps to WriteOptions#withColumnEncoding(ColumnName, ColumnEncoding.candidates(...)), i.e. the whole candidate set for that column, as in Rust. Strings are parsed once at the boundary into EncodingId; an unknown id fails fast.
  • Column name (schema): @Column("price") when the component name is not the column name; useful for projection records over files written by someone else.
  • Later, once the nested and extension types land: @Decimal(precision, scale) on BigDecimal, and the zone-map/stats toggles.

Out of scope

Filters and aggregates as lambdas, async APIs, annotation processing or code generation (the first version can use java.lang.reflect.RecordComponent).

Vorherrschende Sprache
Java
Sterne
3
Forks
2
Ø Merge
9 Std. 47 Min.
Gemergte PRs (30 T.)
128

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus dfa1/vortex-java

Alle Issues in dfa1/vortex-java

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.