pingcap/tidb

Support inline projection for executors

Open

#14,428 建立於 2020年1月9日

在 GitHub 查看
 (28 留言) (4 反應) (2 負責人)Go (40,090 star) (6,186 fork)batch import
challenge-programhelp wantedsig/executionsig/plannertype/enhancement

描述

Description

Is your feature request related to a problem? Please describe:

For example,

select t1.b+1 from t1, t2 where t1.a = t2.a

Join executor reads t1.a and t1.b from table t1, t2.a from table t2, then concatenates and materilizes these three columns. Next step, projection executor projects three columns to one column of t1.b.

Describe the feature you'd like:

We can optimize it by set the output schema of Join executor as only t1.b, so that the executor do know that it only need to output t1.b. It can avoid copying t1.a and t2.a which are not needed by its parent when materilizes/copys data buffers.

Inline Projection

Here's a demostration PR to show how the performance we gain from avoiding copying extra columns for SORT executor.

What we need to do:

Executor Why need this
Sort Copy rows with random access
TopN Copy rows with random access
Join (Hash Join, Merge Join, Index Merge Join, Index Hash Join) Copy rows with random access from build table, join rows from one of the build table and one of the probe table
Apply Copy rows with random access from build table, join rows from one of the build table and one of the probe table
Selection Copy rows with random access
Limit? Prune copying of extra columns
  • Support inline projection in planner @SunRunAway #14683 #14791
  • Implement inline projection for Sort and TopN (750 score) @hidehalo #19900 doing
  • Join
    • Implement inline projection for Hash Join @SunRunAway #14682 #14783
    • Implement inline projection for Merge Join @lance6716 #15463
    • Implement inline projection for Index Merge Join #15004
    • Implement inline projection for Index Hash Join #15004
  • Implement inline projection for Apply (750 score)
  • Implement inline projection Selection (750 score) @pengdaqian2020 #20335 doing
  • Implement inline projection for Limit (750 score) @pingyu https://github.com/pingcap/tidb/pull/20288

Difficulty

  • Medium

Score

  • 3000

Mentor(s)

  • @SunRunAway
  • @pingyu

Recommended Skills

  • SQL Optimization
  • Golang Profiling
  • Code Refactoring

貢獻者指南

技術棧
gosql
領域
databaseperformance
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
over 1 week
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
active
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
Go programmingSQL query executionProfiling tools
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
20
研究方向
Review the demonstration PR #14353 to understand the optimization pattern. Examine implemented examples like Hash Join (#14682) and Limit (#20288). For a specific uncompleted subtask like Apply or Selection, study the executor's code to identify where columns are copied unnecessarily and propose modifications to prune non needed columns from output schema. Engage with mentors @SunRunAway and @pingyu for guidance.