New module `mapper`: read and write Java records as typed rows
维护者通常 1 天内回复
评估
调研方向
Start by reading the existing reader and writer modules, especially VortexReader, VortexWriter, and the ScanOptions and WriteOptions APIs; the issue names no specific file paths or tests. Define a new mapper module that depends on both modules and inspect how their column types and batches are represented. Done means Java records map to the listed basic types, projection reads validate names and types, annotations follow the described rules, and unsupported types fail clearly.
由索引模型根据 Issue 内容生成。
描述
A small module, mapper, that maps a Java record to a Vortex struct schema, for both the reader and the writer. Same idea as the typed-record module we did for typesafe-java; inspired by Vorticity v0.1.0, whose [VortexRecord] declares the columns of a file.
Minimum requirements
- A Java
recordis the schema: its components are the columns, in order, under their names (ColumnName). - Write: records in, a
.vortexfile out, through the existingVortexWriter. Taking aList<Record>is fine and convenient: the usual source is row oriented, so the writer pivots rows into column batches. - Read: a
.vortexfile in, records out, through the existingVortexReader. - A new
mappermodule that depends onreaderandwriter; neither of them depends on it (the writer never depends on the reader today, and this module is the only place that sees both). - An unsupported component type fails fast with a clear error.
Projection is the reading model
A columnar file is read by column, never select *. The read side must make that the obvious path and say so in the docs:
- A record passed to read is a projection: only its components are decoded, so a record with 1 or 2 components reads 1 or 2 columns of a file that has many more (it maps to
ScanOptionscolumns). - The record need not match the file's full schema: any subset of the file's columns, by name, is valid. A component with no matching column, or with a different type, fails fast.
- Document it with an example: a wide file written from a full
Traderecord, read back through a smallPriceOnly(double price)record. Do not show reading the whole schema as the default.
First round: basic types only
Mapped from the Rust DType variants, only the basic ones:
DType |
Java component |
|---|---|
Bool |
boolean / Boolean |
Primitive |
byte, short, int, long, float, double (boxed = nullable); unsigned via @Unsigned (see below); F16 is deferred |
Utf8 |
String |
Binary |
byte[] |
Nullability follows the component type: a boxed type or a reference type is nullable, a Java primitive is not.
F16 is deferred until Valhalla. A short of raw half bits needs an annotation to tell it from I16, and that API would be replaced once the JDK has a half-float value type. Until then a record component cannot map to an F16 column: it fails fast with a clear error, and the array API still reads and writes F16 as short[].
Later rounds, not in this ticket: Decimal, List, FixedSizeList, Map, nested Struct (a record inside a record), Union, Variant, and Extension (dates, times, timestamps, UUID), plus Null.
Exploratory: annotations
The module is exploratory, so these are proposals to try, not commitments. Principle: a schema annotation (changes the DType) must agree on the read and write side and is checked fail-fast; a write-time annotation is ignored when reading.
- Unsigned (schema): Java has no unsigned types, and the existing array API already stores
U8..U64in the samebyte[]/short[]/int[]/long[]asI8..I64. Try@Unsignedon a same-width Java primitive (@Unsigned intisU32), bit-identical with no conversion; the user reads it withInteger.toUnsignedLongand friends. A lossless widening on read (U32aslong) is a possible later addition. - Forcing encodings (write-time):
@Encoding("vortex.zstd", ...)on a component maps toWriteOptions#withColumnEncoding(ColumnName, ColumnEncoding.candidates(...)), i.e. the whole candidate set for that column, as in Rust. Strings are parsed once at the boundary intoEncodingId; an unknown id fails fast. - Column name (schema):
@Column("price")when the component name is not the column name; useful for projection records over files written by someone else. - Later, once the nested and extension types land:
@Decimal(precision, scale)onBigDecimal, and the zone-map/stats toggles.
Out of scope
Filters and aggregates as lambdas, async APIs, annotation processing or code generation (the first version can use java.lang.reflect.RecordComponent).
- 主要语言
- Java
- 星标
- 3
- 派生
- 2
- 平均合并
- 9 小时 47 分钟
- 30 天内合并 PR
- 128
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
dfa1/vortex-java 的其他 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 66/100
dfa1/vortex-java#636 ·
维护者通常 1 天内回复
-
bug
难度 3/5 1-2 天 新手友好度 70/100
dfa1/vortex-java#655 ·
维护者通常 1 天内回复
-
perf
难度 4/5 3-5 天 新手友好度 30/100
dfa1/vortex-java#651 ·
维护者通常 1 天内回复
-
Writer: string columns written as per-chunk dicts where Rust uses its dict layout (up to 2x larger)未关闭perf
难度 4/5 3-5 天 新手友好度 30/100
dfa1/vortex-java#650 ·
维护者通常 1 天内回复
-
perf
难度 4/5 3-5 天 新手友好度 28/100
dfa1/vortex-java#649 ·
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 66/100
a2aproject/a2a-java#1275 ·
维护者通常 1 天内回复
-
module/ftp Type/Bug
难度 2/5 1-3 小时 新手友好度 72/100
ballerina-platform/ballerina-library#9476 ·
维护者通常 1 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
anwilli5/coin-collection-android-US#515 ·
维护者通常 1 天内回复
-
status: waiting-for-triage type: bug
难度 2/5 1-3 小时 新手友好度 72/100
spring-projects/spring-integration#11654 ·
维护者通常 1 天内回复
-
难度 2/5 1 小时以内 新手友好度 70/100
Vakore/ZappierGames#248 ·