pingcap/tidb

Support partition spilling for HashAgg

Open

#26,915 opened on Aug 5, 2021

View on GitHub
 (4 comments) (0 reactions) (1 assignee)Go (40,090 stars) (6,186 forks)batch import
help wantedsig/executiontype/enhancement

Description

Enhancement

Now we support spiling unparallel hashagg by a naive approach: when memory usage is higher than quota, spilling all unprocessed data reading from child executor.

There is a reasonable optimization point: Hash partition the data while spilling data

There are many advantages following:

  • Correctness. The way can keep all data that have the same key will be spilled in the same partition, and processed in the same time soon.
  • Less memory usage. Obviously, processing less data will use less memory.
  • Reduce IO. Now the spilling algorithm maybe re-spilling some data that has been spilled last round when memory usage is higher than quota again. Spilling partition data is always better than spilling full data.

Reference

Contributor guide

Tech stack
go
Domain
databaseperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Go programmingDatabase query executionHash aggregationMemory management
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Study the design doc for spilling HashAgg at docs/design/2021-06-23 spilled unparallel hashagg.md and the current implementation in executor/aggregate.go. Understand the naive approach and analyze how partition spilling could be implemented. Review any existing comments in the issue (4 comments) for discussion points.