use external storage to reduce the memory space usage of some executors · pingcap/tidb#14392

(1 comment) (0 reactions) (0 assignees)Go (40,090 stars) (6,186 forks)batch import

epic/memory-managementhelp wantedsig/executiontype/enhancement

説明

Is your feature request related to a problem? Please describe:

As the title says.

Describe the feature you'd like:

issue	difficulty	assignee	PRs
[ ] #TODO Use external storage for Index Merge Join	hard	help wanted	N/A
[ ] #14675 implement grace hash join for Hash Join to avoid OOM of hash table	hard	help wanted	N/A
[ ] #14413 Use external storage for Hash Aggregation	hard	help wanted	N/A
[ ] #14395 Use external storage for Distinct	hard	help wanted	N/A
[x] #12431 Use external storage for Sort	medium	@wshwsh12	#13718, #14279
[ ] #TODO Use external storage for Window Function	medium	help wanted	N/A
[ ] #TODO Compare between row-based and vectorized external storage format for different executors	hard	help wanted	N/A

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

技術スタック: go
領域: databaseperformance
Issue 種別: feature
難度: 5
推定時間: over 1 week
活動状況: stale
明確さ: mostly clear
前提条件: Godatabase internalsmemory management
初心者向け度: 15
調査方針: Start by reviewing the existing Sort executor external storage implementation (issue #12431, PRs #13718, #14279) to understand the approach. Then examine the codebase for executors like Index Merge Join, Hash Join, and Hash Aggregation to identify memory usage hotspots. Look for storage interfaces in the TiDB source code and consider how to integrate external storage to spill data to disk when memory limits are exceeded. The issue includes multiple sub tasks with difficulty ratings; focus on one executor with 'help wanted' status.