lance-format/lance

Allow dictionary encoding to support more fixed width types

Open

#5,004 opened on 2025年10月20日

GitHub で見る
 (2 comments) (0 reactions) (0 assignees)Rust (695 forks)github user discovery
good first issue

Repository metrics

Stars
 (6,582 stars)
PR merge metrics
 (平均マージ 6d 1h) (30d で 219 merged PRs)

説明

Currently the dictionary encoding in 2.1 only applies to some 128-bit fixed-width types. I think this was a combination of a few things:

  • At the time, we were not bit-packing indices, and so applying dictionary encoding to smaller types was unlikely to be beneficial
  • Estimating cardinality on all columns was expensive
  • TPC-H data has a lot of 128-bit decimals that benefit from dictionary encoding so there was an outsized (and unrealistic) benchmarking gain to focusing on 128-bit types.

However, we should at least be considering dictionary encoding for 64-bit types and maybe even 32-bit types.

コントリビューターガイド