Writer: flat array stats should match the ones Rust keeps per encoding node

オープン
  • 何か問題がありますか？
#583 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
30/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
java, rust
領域
backend, database

調査の方向性

Start by reading flat/writer.rs and the listed encoding files in the Rust source to trace which statistics each encoding inherits and retains. Then inspect Java's flat writer and cascade, along with the vortex-jni integration tests, to compare stats on every node for the listed column shapes. Done means the per-node stats match between writers, with the string min/max and null-count expectations updated where Rust omits them.

索引モデルが issue の本文から書いたものです。

説明

Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.

What Rust does

FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).

Observed (same 20k-row data through both writers, all stats Exact)

root encoding (Rust) Rust writes Java writes
vortex.primitive, fastlanes.bitpacked min, max, is_constant=false, null_count min, max, null_count
vortex.sequence is_sorted, is_strict_sorted min, max, null_count
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant nothing min, max, null_count
vortex.bool is_constant min, max, null_count
child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) often stats (sorted flags, min/max/sum/null_count) never: Java writes stats on the root only

Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.

Plan

  1. Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
  2. Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.

Expected consequences (intended, per the parity directive)

  • Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So VortexReader.columnStats() string min/max and the SQL MIN/MAX stats shortcut go away for them, as already happens for Rust-written files. JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds (and the emoji test above it) then expect null min/max.
  • Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
  • Numeric columnStats are unaffected, because they come from the zone table.
  • VortexWriter.buildArrayFlatBuffer's comment says null_count is always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.

The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.

主要言語
Java
スター
3
フォーク
2
平均マージ
9時間 47分
マージ済み PR（30日）
128

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dfa1/vortex-java のほかの issue

dfa1/vortex-java の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。