Writer: numeric dict layout is chosen by a 2,048-distinct cap; Rust probes and writes several dictionaries

Aberta
  • Algo errado?
#649 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
28/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
java, rust
Domínio
data

Direção de pesquisa

Start in DictColumnState.isDictCandidate and the GLOBAL_DICT_MAX_CARDINALITY cap in vortex-java, then compare with DictStrategy in vortex-layout/src/layouts/dict/writer.rs, which probes a sample and writes several dictionaries. Read ADR 0021 before changing the heuristic. Reproduce with Int64NarrowingInteropIntegrationTest using a one-column HASH_CSV dump. Done means the RefererHash tree matches the vortex-jni dict layout and high-cardinality I64/F64 columns show no regressions.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

perf

Raincloud clickbench-hits, vortex-java's Parquet import vs the vortex-jni-written file, data bytes (99.997M rows):

column type Rust Java
RefererHash I64 401.0 MB 791.3 MB (1.97x)
URLHash I64 427.3 MB 790.0 MB (1.85x)

Cause (confirmed)

Not integer narrowing: the values are full 64-bit. Rust writes these columns as a dict layout; Java writes plain vortex.primitive (8 bytes a row).

Reproduced with 131,072 real RefererHash rows (first chunk of ClickBench hits.parquet, ~23k distinct), written by both writers:

file tree
vortex-jni 434 KB vortex.dict layout: vortex.primitive values + fastlanes.bitpacked codes
vortex-java 1,050 KB vortex.primitive

Synthetic i32/u32/i16-range, mixed and full-i64 columns already produce identical trees in both writers, so narrowing is not involved.

  • Rust (vortex-layout/src/layouts/dict/writer.rs, DictStrategy): compresses a probe sample with the cascade; if the result is a Dict, it writes a dict layout, up to u16::MAX values per dictionary, and starts a new dictionary when the constraint is hit.
  • Java (DictColumnState.isDictCandidate): a numeric column is offered a global dict only with at most GLOBAL_DICT_MAX_CARDINALITY (2,048) distinct values and under 50% distinct, one dictionary for the whole column, demoted to per-chunk encoding past the cap. This column has ~23k distinct per chunk and ~719k over 2M rows.

Fix

Raising the cap is not enough: over the whole column Rust writes several dictionaries in sequence. Parity means replacing the cardinality heuristic with Rust's probe-then-multi-dictionary flow (touches ADR 0021, and the comment on the cap says a global dict hurt some high-cardinality I64/F64 columns, so check those for regressions).

Repro: Int64NarrowingInteropIntegrationTest (exploratory, not committed), fed HASH_CSV=<one-column dump>.

Linguagem predominante
Java
Estrelas
3
Forks
2
Merge médio
9h 47min
PRs com merge (30d)
128

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de dfa1/vortex-java

Todas as issues de dfa1/vortex-java

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.