Writer: flat array stats should match the ones Rust keeps per encoding node

Ouverte
  • Quelque chose ne va pas ?
#583 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
30/100
Type d'issue
Fonctionnalité
Clarté
Clairement spécifiée
Activité
Active
Stack technique
java, rust
Domaine
backend, database

Piste de recherche

Start by reading flat/writer.rs and the listed encoding files in the Rust source to trace which statistics each encoding inherits and retains. Then inspect Java's flat writer and cascade, along with the vortex-jni integration tests, to compare stats on every node for the listed column shapes. Done means the per-node stats match between writers, with the string min/max and null-count expectations updated where Rust omits them.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.

What Rust does

FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).

Observed (same 20k-row data through both writers, all stats Exact)

root encoding (Rust) Rust writes Java writes
vortex.primitive, fastlanes.bitpacked min, max, is_constant=false, null_count min, max, null_count
vortex.sequence is_sorted, is_strict_sorted min, max, null_count
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant nothing min, max, null_count
vortex.bool is_constant min, max, null_count
child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) often stats (sorted flags, min/max/sum/null_count) never: Java writes stats on the root only

Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.

Plan

  1. Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
  2. Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.

Expected consequences (intended, per the parity directive)

  • Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So VortexReader.columnStats() string min/max and the SQL MIN/MAX stats shortcut go away for them, as already happens for Rust-written files. JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds (and the emoji test above it) then expect null min/max.
  • Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
  • Numeric columnStats are unaffected, because they come from the zone table.
  • VortexWriter.buildArrayFlatBuffer's comment says null_count is always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.

The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.

Langage dominant
Java
Étoiles
3
Forks
2
Merge moyen
14 h 59 min
PR mergées (30 j)
137

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de dfa1/vortex-java

Toutes les issues de dfa1/vortex-java

Issues similaires

Plus d'issues Java

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.