pingcap/tidb

use external storage to reduce the memory space usage of some executors

Open

#14,392 建立於 2020年1月8日

在 GitHub 查看
 (1 留言) (0 反應) (0 負責人)Go (40,090 star) (6,186 fork)batch import
epic/memory-managementhelp wantedsig/executiontype/enhancement

描述

Feature Request

Is your feature request related to a problem? Please describe:

As the title says.

Describe the feature you'd like:

issue difficulty assignee PRs
[ ] #TODO Use external storage for Index Merge Join hard help wanted N/A
[ ] #14675 implement grace hash join for Hash Join to avoid OOM of hash table hard help wanted N/A
[ ] #14413 Use external storage for Hash Aggregation hard help wanted N/A
[ ] #14395 Use external storage for Distinct hard help wanted N/A
[x] #12431 Use external storage for Sort medium @wshwsh12 #13718, #14279
[ ] #TODO Use external storage for Window Function medium help wanted N/A
[ ] #TODO Compare between row-based and vectorized external storage format for different executors hard help wanted N/A

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

貢獻者指南

技術棧
go
領域
databaseperformance
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
5
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
over 1 week
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
Godatabase internalsmemory management
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
15
研究方向
Start by reviewing the existing Sort executor external storage implementation (issue #12431, PRs #13718, #14279) to understand the approach. Then examine the codebase for executors like Index Merge Join, Hash Join, and Hash Aggregation to identify memory usage hotspots. Look for storage interfaces in the TiDB source code and consider how to integrate external storage to spill data to disk when memory limits are exceeded. The issue includes multiple sub tasks with difficulty ratings; focus on one executor with 'help wanted' status.