lance-format/lance
GitHub で見るAllow dictionary encoding to support more fixed width types
Open
#5,004 opened on 2025年10月20日
good first issue
Repository metrics
- Stars
- (6,582 stars)
- PR merge metrics
- (平均マージ 6d 1h) (30d で 219 merged PRs)
説明
Currently the dictionary encoding in 2.1 only applies to some 128-bit fixed-width types. I think this was a combination of a few things:
- At the time, we were not bit-packing indices, and so applying dictionary encoding to smaller types was unlikely to be beneficial
- Estimating cardinality on all columns was expensive
- TPC-H data has a lot of 128-bit decimals that benefit from dictionary encoding so there was an outsized (and unrealistic) benchmarking gain to focusing on 128-bit types.
However, we should at least be considering dictionary encoding for 64-bit types and maybe even 32-bit types.