Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries
維護者通常 1 天內回覆
還沒有人認領這個 Issue。
評估
研究方向
Start in DictColumnState.isDictCandidate and the GLOBAL_DICT_MAX_CARDINALITY cap in vortex-java, then compare with DictStrategy in vortex-layout/src/layouts/dict/writer.rs, which probes a sample and writes several dictionaries. Read ADR 0021 before changing the heuristic. Reproduce with Int64NarrowingInteropIntegrationTest using a one-column HASH_CSV dump. Done means the RefererHash tree matches the vortex-jni dict layout and high-cardinality I64/F64 columns show no regressions.
由索引模型根據 Issue 內容生成。
描述
Raincloud clickbench-hits, vortex-java's Parquet import vs the vortex-jni-written file, data bytes (99.997M rows):
| column | type | Rust | Java |
|---|---|---|---|
| RefererHash | I64 | 401.0 MB | 791.3 MB (1.97x) |
| URLHash | I64 | 427.3 MB | 790.0 MB (1.85x) |
Cause (confirmed)
Not integer narrowing: the values are full 64-bit. Rust writes these columns as a dict layout; Java writes plain vortex.primitive (8 bytes a row).
Reproduced with 131,072 real RefererHash rows (first chunk of ClickBench hits.parquet, ~23k distinct), written by both writers:
| file | tree | |
|---|---|---|
| vortex-jni | 434 KB | vortex.dict layout: vortex.primitive values + fastlanes.bitpacked codes |
| vortex-java | 1,050 KB | vortex.primitive |
Synthetic i32/u32/i16-range, mixed and full-i64 columns already produce identical trees in both writers, so narrowing is not involved.
- Rust (
vortex-layout/src/layouts/dict/writer.rs,DictStrategy): compresses a probe sample with the cascade; if the result is aDict, it writes a dict layout, up tou16::MAXvalues per dictionary, and starts a new dictionary when the constraint is hit. - Java (
DictColumnState.isDictCandidate): a numeric column is offered a global dict only with at mostGLOBAL_DICT_MAX_CARDINALITY(2,048) distinct values and under 50% distinct, one dictionary for the whole column, demoted to per-chunk encoding past the cap. This column has ~23k distinct per chunk and ~719k over 2M rows.
Fix
Raising the cap is not enough: over the whole column Rust writes several dictionaries in sequence. Parity means replacing the cardinality heuristic with Rust's probe-then-multi-dictionary flow (touches ADR 0021, and the comment on the cap says a global dict hurt some high-cardinality I64/F64 columns, so check those for regressions).
Repro: Int64NarrowingInteropIntegrationTest (exploratory, not committed), fed HASH_CSV=<one-column dump>.
- 主要語言
- Java
- 星號
- 3
- 分支
- 2
- 平均合併
- 9 小時 55 分鐘
- 30 天內合併 PR
- 126
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
dfa1/vortex-java 的其他 Issue
-
bug
難度 2/5 1-3 小時 新手友好度 66/100
dfa1/vortex-java#636 ·
維護者通常 1 天內回覆
-
enhancement
難度 4/5 3-5 天 新手友好度 38/100
dfa1/vortex-java#660 ·
維護者通常 1 天內回覆
-
bug
難度 4/5 3-5 天 新手友好度 35/100
dfa1/vortex-java#657 · 1 則留言 ·
維護者通常 1 天內回覆
-
bug
難度 3/5 1-2 天 新手友好度 70/100
dfa1/vortex-java#655 ·
維護者通常 1 天內回覆
-
perf
難度 4/5 3-5 天 新手友好度 30/100
dfa1/vortex-java#651 ·
維護者通常 1 天內回覆
相似的 Issue
-
vaadin-map
難度 2/5 1-3 小時 新手友好度 70/100
vaadin/flow-components#10586 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 63/100
wpilibsuite/SystemcoreTesting#446 ·
維護者通常 1 天內回覆
-
frontend quality-assurance
難度 2/5 1-3 小時 新手友好度 64/100
Paul-Austin-Oswego-CSC480-HCI521/gift-app#212 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 66/100
eip-work/kuboard-press#700 ·
-
Mend: dependency security vulnerability
難度 2/5 1-3 小時 新手友好度 65/100
維護者通常 1 天內回覆