Writer: flat array stats should match the ones Rust keeps per encoding node

Offen
  • Stimmt etwas nicht?
#583 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
30/100
Issue-Typ
Feature
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
java, rust
Bereich
backend, database

Rechercherichtung

Start by reading flat/writer.rs and the listed encoding files in the Rust source to trace which statistics each encoding inherits and retains. Then inspect Java's flat writer and cascade, along with the vortex-jni integration tests, to compare stats on every node for the listed column shapes. Done means the per-node stats match between writers, with the string min/max and null-count expectations updated where Rust omits them.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.

What Rust does

FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).

Observed (same 20k-row data through both writers, all stats Exact)

root encoding (Rust) Rust writes Java writes
vortex.primitive, fastlanes.bitpacked min, max, is_constant=false, null_count min, max, null_count
vortex.sequence is_sorted, is_strict_sorted min, max, null_count
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant nothing min, max, null_count
vortex.bool is_constant min, max, null_count
child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) often stats (sorted flags, min/max/sum/null_count) never: Java writes stats on the root only

Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.

Plan

  1. Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
  2. Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.

Expected consequences (intended, per the parity directive)

  • Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So VortexReader.columnStats() string min/max and the SQL MIN/MAX stats shortcut go away for them, as already happens for Rust-written files. JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds (and the emoji test above it) then expect null min/max.
  • Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
  • Numeric columnStats are unaffected, because they come from the zone table.
  • VortexWriter.buildArrayFlatBuffer's comment says null_count is always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.

The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.

Vorherrschende Sprache
Java
Sterne
3
Forks
2
Ø Merge
14 Std. 59 Min.
Gemergte PRs (30 T.)
137

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus dfa1/vortex-java

Alle Issues in dfa1/vortex-java

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.