Writer: flat array stats should match the ones Rust keeps per encoding node

Abierto
  • ¿Algo no está bien?
#583 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
java, rust
Área
backend, database

Línea de trabajo

Start by reading flat/writer.rs and the listed encoding files in the Rust source to trace which statistics each encoding inherits and retains. Then inspect Java's flat writer and cascade, along with the vortex-jni integration tests, to compare stats on every node for the listed column shapes. Done means the per-node stats match between writers, with the string min/max and null-count expectations updated where Rust omits them.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.

What Rust does

FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).

Observed (same 20k-row data through both writers, all stats Exact)

root encoding (Rust) Rust writes Java writes
vortex.primitive, fastlanes.bitpacked min, max, is_constant=false, null_count min, max, null_count
vortex.sequence is_sorted, is_strict_sorted min, max, null_count
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant nothing min, max, null_count
vortex.bool is_constant min, max, null_count
child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) often stats (sorted flags, min/max/sum/null_count) never: Java writes stats on the root only

Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.

Plan

  1. Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
  2. Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.

Expected consequences (intended, per the parity directive)

  • Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So VortexReader.columnStats() string min/max and the SQL MIN/MAX stats shortcut go away for them, as already happens for Rust-written files. JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds (and the emoji test above it) then expect null min/max.
  • Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
  • Numeric columnStats are unaffected, because they come from the zone table.
  • VortexWriter.buildArrayFlatBuffer's comment says null_count is always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.

The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.

Lenguaje dominante
Java
Estrellas
3
Forks
2
Merge medio
9 h 47 min
PR fusionados (30 d)
128

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dfa1/vortex-java

Todos los issues de dfa1/vortex-java

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.