pingcap/tidb

Parallel evaluation for aggregate functions with distinct in hashAgg

Open

#17,075 opened on May 10, 2020

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Go (40,090 stars) (6,186 forks)batch import
help wantedsig/executiontype/enhancementtype/performance

Description

Development Task

select count(distinct a) from t;

Currently if one of the aggregate functions of a SQL has a distinct evaluation, the hashAgg executor must run in a single thread. See https://github.com/pingcap/tidb/blob/3224393/executor/builder.go#L1207

We may find a method to evaluate it in parallel, maybe a sync.Map for deduplicating entries can be shared in multiple goroutines.

Contributor guide

Tech stack
go
Domain
backenddatabaseperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Goparallelismsync.Map
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Examine the current implementation in executor/builder.go around line 1207. The approach involves using sync.Map for deduplication across goroutines. The main challenge is ensuring thread safe accumulation while maintaining correctness. Refer to the linked file and consider implementing a shared sync.Map for distinct values, then aggregating results after goroutines finish. Test with the provided SQL query.