pingcap/tidb

Support partition spilling for HashAgg

Open

#26,915 创建于 2021年8月5日

在 GitHub 查看
 (4 评论) (0 反应) (1 负责人)Go (6,186 fork)batch import
help wantedsig/executiontype/enhancement

仓库指标

Star
 (40,090 star)
PR 合并指标
 (平均合并 14天 4小时) (30 天内合并 346 个 PR)

描述

Enhancement

Now we support spiling unparallel hashagg by a naive approach: when memory usage is higher than quota, spilling all unprocessed data reading from child executor.

There is a reasonable optimization point: Hash partition the data while spilling data

There are many advantages following:

  • Correctness. The way can keep all data that have the same key will be spilled in the same partition, and processed in the same time soon.
  • Less memory usage. Obviously, processing less data will use less memory.
  • Reduce IO. Now the spilling algorithm maybe re-spilling some data that has been spilled last round when memory usage is higher than quota again. Spilling partition data is always better than spilling full data.

Reference

贡献者指南