New module `mapper`: read and write Java records as typed rows

Abierto
  • ¿Algo no está bien?
#528 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@dfa1 ya está trabajando en esto.

Desde el 17/10/2026.

  • #549 de @dfa1 — abierto

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
java
Área
database

Línea de trabajo

Start by reading the existing reader and writer modules, especially VortexReader, VortexWriter, and the ScanOptions and WriteOptions APIs; the issue names no specific file paths or tests. Define a new mapper module that depends on both modules and inspect how their column types and batches are represented. Done means Java records map to the listed basic types, projection reads validate names and types, annotations follow the described rules, and unsupported types fail clearly.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement help wanted PARKED

A small module, mapper, that maps a Java record to a Vortex struct schema, for both the reader and the writer. Same idea as the typed-record module we did for typesafe-java; inspired by Vorticity v0.1.0, whose [VortexRecord] declares the columns of a file.

Minimum requirements

  • A Java record is the schema: its components are the columns, in order, under their names (ColumnName).
  • Write: records in, a .vortex file out, through the existing VortexWriter. Taking a List<Record> is fine and convenient: the usual source is row oriented, so the writer pivots rows into column batches.
  • Read: a .vortex file in, records out, through the existing VortexReader.
  • A new mapper module that depends on reader and writer; neither of them depends on it (the writer never depends on the reader today, and this module is the only place that sees both).
  • An unsupported component type fails fast with a clear error.

Projection is the reading model

A columnar file is read by column, never select *. The read side must make that the obvious path and say so in the docs:

  • A record passed to read is a projection: only its components are decoded, so a record with 1 or 2 components reads 1 or 2 columns of a file that has many more (it maps to ScanOptions columns).
  • The record need not match the file's full schema: any subset of the file's columns, by name, is valid. A component with no matching column, or with a different type, fails fast.
  • Document it with an example: a wide file written from a full Trade record, read back through a small PriceOnly(double price) record. Do not show reading the whole schema as the default.

First round: basic types only

Mapped from the Rust DType variants, only the basic ones:

DType Java component
Bool boolean / Boolean
Primitive byte, short, int, long, float, double (boxed = nullable); unsigned via @Unsigned (see below); F16 is deferred
Utf8 String
Binary byte[]

Nullability follows the component type: a boxed type or a reference type is nullable, a Java primitive is not.

F16 is deferred until Valhalla. A short of raw half bits needs an annotation to tell it from I16, and that API would be replaced once the JDK has a half-float value type. Until then a record component cannot map to an F16 column: it fails fast with a clear error, and the array API still reads and writes F16 as short[].

Later rounds, not in this ticket: Decimal, List, FixedSizeList, Map, nested Struct (a record inside a record), Union, Variant, and Extension (dates, times, timestamps, UUID), plus Null.

Exploratory: annotations

The module is exploratory, so these are proposals to try, not commitments. Principle: a schema annotation (changes the DType) must agree on the read and write side and is checked fail-fast; a write-time annotation is ignored when reading.

  • Unsigned (schema): Java has no unsigned types, and the existing array API already stores U8..U64 in the same byte[]/short[]/int[]/long[] as I8..I64. Try @Unsigned on a same-width Java primitive (@Unsigned int is U32), bit-identical with no conversion; the user reads it with Integer.toUnsignedLong and friends. A lossless widening on read (U32 as long) is a possible later addition.
  • Forcing encodings (write-time): @Encoding("vortex.zstd", ...) on a component maps to WriteOptions#withColumnEncoding(ColumnName, ColumnEncoding.candidates(...)), i.e. the whole candidate set for that column, as in Rust. Strings are parsed once at the boundary into EncodingId; an unknown id fails fast.
  • Column name (schema): @Column("price") when the component name is not the column name; useful for projection records over files written by someone else.
  • Later, once the nested and extension types land: @Decimal(precision, scale) on BigDecimal, and the zone-map/stats toggles.

Out of scope

Filters and aggregates as lambdas, async APIs, annotation processing or code generation (the first version can use java.lang.reflect.RecordComponent).

Lenguaje dominante
Java
Estrellas
3
Forks
2
Merge medio
9 h 47 min
PR fusionados (30 d)
128

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dfa1/vortex-java

Todos los issues de dfa1/vortex-java

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.