Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries

Aperta
  • Qualcosa non va?
#649 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
28/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, rust
Ambito
data

Direzione di ricerca

Start in DictColumnState.isDictCandidate and the GLOBAL_DICT_MAX_CARDINALITY cap in vortex-java, then compare with DictStrategy in vortex-layout/src/layouts/dict/writer.rs, which probes a sample and writes several dictionaries. Read ADR 0021 before changing the heuristic. Reproduce with Int64NarrowingInteropIntegrationTest using a one-column HASH_CSV dump. Done means the RefererHash tree matches the vortex-jni dict layout and high-cardinality I64/F64 columns show no regressions.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

perf

Raincloud clickbench-hits, vortex-java's Parquet import vs the vortex-jni-written file, data bytes (99.997M rows):

column type Rust Java
RefererHash I64 401.0 MB 791.3 MB (1.97x)
URLHash I64 427.3 MB 790.0 MB (1.85x)

Cause (confirmed)

Not integer narrowing: the values are full 64-bit. Rust writes these columns as a dict layout; Java writes plain vortex.primitive (8 bytes a row).

Reproduced with 131,072 real RefererHash rows (first chunk of ClickBench hits.parquet, ~23k distinct), written by both writers:

file tree
vortex-jni 434 KB vortex.dict layout: vortex.primitive values + fastlanes.bitpacked codes
vortex-java 1,050 KB vortex.primitive

Synthetic i32/u32/i16-range, mixed and full-i64 columns already produce identical trees in both writers, so narrowing is not involved.

  • Rust (vortex-layout/src/layouts/dict/writer.rs, DictStrategy): compresses a probe sample with the cascade; if the result is a Dict, it writes a dict layout, up to u16::MAX values per dictionary, and starts a new dictionary when the constraint is hit.
  • Java (DictColumnState.isDictCandidate): a numeric column is offered a global dict only with at most GLOBAL_DICT_MAX_CARDINALITY (2,048) distinct values and under 50% distinct, one dictionary for the whole column, demoted to per-chunk encoding past the cap. This column has ~23k distinct per chunk and ~719k over 2M rows.

Fix

Raising the cap is not enough: over the whole column Rust writes several dictionaries in sequence. Parity means replacing the cardinality heuristic with Rust's probe-then-multi-dictionary flow (touches ADR 0021, and the comment on the cap says a global dict hurt some high-cardinality I64/F64 columns, so check those for regressions).

Repro: Int64NarrowingInteropIntegrationTest (exploratory, not committed), fed HASH_CSV=<one-column dump>.

Lingua principale
Java
Stelle
3
Fork
2
Merge medio
9h 47m
PR unite (30g)
128

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dfa1/vortex-java

Tutte le issue di dfa1/vortex-java

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.