ClickHouse/ClickHouse

Optimize search in pre-ordered arrays

開放

#17,795 建立於 2020年12月4日

 (13 則留言) (5 個反應) (0 位負責人)C++ (8,400 個分叉)batch import
featurehelp wanted

倉庫指標

星標
 (47,419 顆星)
PR 合併指標
 (平均合併 2天 2小時) (30 天內合併 1,000 個 PR)

描述

Use case In our principal table, we use the type Array(UInt64) for stocking the corresponding ids for a row. Each array can contain thousands of ids, and multiplying it to milliards of rows, we have a very heavy column that hardly limits the performance when we doing selects with WHERE has(ids, 123).

Describe the solution you'd like One solution that I can imagine is to pre-sort the ids in each row and, knowing that all arrays in the column are sorted, we can use it for a small optimization. Concretely, when we doing the search in the ordered array, we are not forced to scan the array entirely, but only before the current_value <= searched_value I estimate that it can be 2x faster in average. And more in special cases.

貢獻者指南