lance-format/lance
在 GitHub 查看Allow dictionary encoding to support more fixed width types
Open
#5,004 建立於 2025年10月20日
good first issue
倉庫指標
- Star
- (6,582 star)
- PR 合併指標
- (平均合併 6天 1小時) (30 天內合併 219 個 PR)
描述
Currently the dictionary encoding in 2.1 only applies to some 128-bit fixed-width types. I think this was a combination of a few things:
- At the time, we were not bit-packing indices, and so applying dictionary encoding to smaller types was unlikely to be beneficial
- Estimating cardinality on all columns was expensive
- TPC-H data has a lot of 128-bit decimals that benefit from dictionary encoding so there was an outsized (and unrealistic) benchmarking gain to focusing on 128-bit types.
However, we should at least be considering dictionary encoding for 64-bit types and maybe even 32-bit types.