Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start in DictColumnState.isDictCandidate and the GLOBAL_DICT_MAX_CARDINALITY cap in vortex-java, then compare with DictStrategy in vortex-layout/src/layouts/dict/writer.rs, which probes a sample and writes several dictionaries. Read ADR 0021 before changing the heuristic. Reproduce with Int64NarrowingInteropIntegrationTest using a one-column HASH_CSV dump. Done means the RefererHash tree matches the vortex-jni dict layout and high-cardinality I64/F64 columns show no regressions.
由索引模型根据 Issue 内容生成。
描述
Raincloud clickbench-hits, vortex-java's Parquet import vs the vortex-jni-written file, data bytes (99.997M rows):
| column | type | Rust | Java |
|---|---|---|---|
| RefererHash | I64 | 401.0 MB | 791.3 MB (1.97x) |
| URLHash | I64 | 427.3 MB | 790.0 MB (1.85x) |
Cause (confirmed)
Not integer narrowing: the values are full 64-bit. Rust writes these columns as a dict layout; Java writes plain vortex.primitive (8 bytes a row).
Reproduced with 131,072 real RefererHash rows (first chunk of ClickBench hits.parquet, ~23k distinct), written by both writers:
| file | tree | |
|---|---|---|
| vortex-jni | 434 KB | vortex.dict layout: vortex.primitive values + fastlanes.bitpacked codes |
| vortex-java | 1,050 KB | vortex.primitive |
Synthetic i32/u32/i16-range, mixed and full-i64 columns already produce identical trees in both writers, so narrowing is not involved.
- Rust (
vortex-layout/src/layouts/dict/writer.rs,DictStrategy): compresses a probe sample with the cascade; if the result is aDict, it writes a dict layout, up tou16::MAXvalues per dictionary, and starts a new dictionary when the constraint is hit. - Java (
DictColumnState.isDictCandidate): a numeric column is offered a global dict only with at mostGLOBAL_DICT_MAX_CARDINALITY(2,048) distinct values and under 50% distinct, one dictionary for the whole column, demoted to per-chunk encoding past the cap. This column has ~23k distinct per chunk and ~719k over 2M rows.
Fix
Raising the cap is not enough: over the whole column Rust writes several dictionaries in sequence. Parity means replacing the cardinality heuristic with Rust's probe-then-multi-dictionary flow (touches ADR 0021, and the comment on the cap says a global dict hurt some high-cardinality I64/F64 columns, so check those for regressions).
Repro: Int64NarrowingInteropIntegrationTest (exploratory, not committed), fed HASH_CSV=<one-column dump>.
- 主要语言
- Java
- 星标
- 3
- 派生
- 2
- 平均合并
- 9 小时 51 分钟
- 30 天内合并 PR
- 127
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
dfa1/vortex-java 的其他 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 66/100
dfa1/vortex-java#636 ·
维护者通常 1 天内回复
-
bug
难度 4/5 3-5 天 新手友好度 35/100
dfa1/vortex-java#657 ·
维护者通常 1 天内回复
-
bug
难度 3/5 1-2 天 新手友好度 70/100
dfa1/vortex-java#655 ·
维护者通常 1 天内回复
-
perf
难度 4/5 3-5 天 新手友好度 30/100
dfa1/vortex-java#651 ·
维护者通常 1 天内回复
-
Writer: string columns written as per-chunk dicts where Rust uses its dict layout (up to 2x larger)未关闭perf
难度 4/5 3-5 天 新手友好度 30/100
dfa1/vortex-java#650 ·
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
Dans-Plugins/Medieval-Roleplay-Engine#396 ·
维护者通常 1 天内回复
-
bug
难度 1/5 1 小时以内 新手友好度 82/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
objectionary/lints#1640 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 72/100
virtualcell/vcell#2193 ·
维护者通常 1 天内回复