New module `mapper`: read and write Java records as typed rows

オープン
  • 何か問題がありますか？
#528 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@dfa1 がすでに取り組んでいます。

2026年10月17日 から。

  • #549 @dfa1 による — オープン

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
java
領域
database

調査の方向性

Start by reading the existing reader and writer modules, especially VortexReader, VortexWriter, and the ScanOptions and WriteOptions APIs; the issue names no specific file paths or tests. Define a new mapper module that depends on both modules and inspect how their column types and batches are represented. Done means Java records map to the listed basic types, projection reads validate names and types, annotations follow the described rules, and unsupported types fail clearly.

索引モデルが issue の本文から書いたものです。

説明

enhancement help wanted PARKED

A small module, mapper, that maps a Java record to a Vortex struct schema, for both the reader and the writer. Same idea as the typed-record module we did for typesafe-java; inspired by Vorticity v0.1.0, whose [VortexRecord] declares the columns of a file.

Minimum requirements

  • A Java record is the schema: its components are the columns, in order, under their names (ColumnName).
  • Write: records in, a .vortex file out, through the existing VortexWriter. Taking a List<Record> is fine and convenient: the usual source is row oriented, so the writer pivots rows into column batches.
  • Read: a .vortex file in, records out, through the existing VortexReader.
  • A new mapper module that depends on reader and writer; neither of them depends on it (the writer never depends on the reader today, and this module is the only place that sees both).
  • An unsupported component type fails fast with a clear error.

Projection is the reading model

A columnar file is read by column, never select *. The read side must make that the obvious path and say so in the docs:

  • A record passed to read is a projection: only its components are decoded, so a record with 1 or 2 components reads 1 or 2 columns of a file that has many more (it maps to ScanOptions columns).
  • The record need not match the file's full schema: any subset of the file's columns, by name, is valid. A component with no matching column, or with a different type, fails fast.
  • Document it with an example: a wide file written from a full Trade record, read back through a small PriceOnly(double price) record. Do not show reading the whole schema as the default.

First round: basic types only

Mapped from the Rust DType variants, only the basic ones:

DType Java component
Bool boolean / Boolean
Primitive byte, short, int, long, float, double (boxed = nullable); unsigned via @Unsigned (see below); F16 is deferred
Utf8 String
Binary byte[]

Nullability follows the component type: a boxed type or a reference type is nullable, a Java primitive is not.

F16 is deferred until Valhalla. A short of raw half bits needs an annotation to tell it from I16, and that API would be replaced once the JDK has a half-float value type. Until then a record component cannot map to an F16 column: it fails fast with a clear error, and the array API still reads and writes F16 as short[].

Later rounds, not in this ticket: Decimal, List, FixedSizeList, Map, nested Struct (a record inside a record), Union, Variant, and Extension (dates, times, timestamps, UUID), plus Null.

Exploratory: annotations

The module is exploratory, so these are proposals to try, not commitments. Principle: a schema annotation (changes the DType) must agree on the read and write side and is checked fail-fast; a write-time annotation is ignored when reading.

  • Unsigned (schema): Java has no unsigned types, and the existing array API already stores U8..U64 in the same byte[]/short[]/int[]/long[] as I8..I64. Try @Unsigned on a same-width Java primitive (@Unsigned int is U32), bit-identical with no conversion; the user reads it with Integer.toUnsignedLong and friends. A lossless widening on read (U32 as long) is a possible later addition.
  • Forcing encodings (write-time): @Encoding("vortex.zstd", ...) on a component maps to WriteOptions#withColumnEncoding(ColumnName, ColumnEncoding.candidates(...)), i.e. the whole candidate set for that column, as in Rust. Strings are parsed once at the boundary into EncodingId; an unknown id fails fast.
  • Column name (schema): @Column("price") when the component name is not the column name; useful for projection records over files written by someone else.
  • Later, once the nested and extension types land: @Decimal(precision, scale) on BigDecimal, and the zone-map/stats toggles.

Out of scope

Filters and aggregates as lambdas, async APIs, annotation processing or code generation (the first version can use java.lang.reflect.RecordComponent).

主要言語
Java
スター
3
フォーク
2
平均マージ
9時間 47分
マージ済み PR（30日）
128

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dfa1/vortex-java のほかの issue

dfa1/vortex-java の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。