pingcap/tidb

improve the executor runtime information collection

Open

#18,663 opened on Jul 17, 2020

View on GitHub
 (2 comments) (3 reactions) (0 assignees)Go (6,186 forks)batch import
help wantedsig/executiontype/enhancement

Repository metrics

Stars
 (40,090 stars)
PR merge metrics
 (Avg merge 14d 4h) (346 merged PRs in 30d)

Description

Current Problem

Currently, from the explain analyze result, we can see some executor is slow, but it's hard to know the reason of the slow.

Development Task

Here is something need to do:

  • refactor execdetails information of runtime collect https://github.com/pingcap/tidb/pull/18530
  • Add more runtime information for Cop task, include backoff_time, backoff_time resolve_lock time, and so on. #20492
  • add runtime stats for below executor:
    • Insert #20430
    • Update #20430
    • Replace #20430
    • Point-Get #18666
    • Batch_Point_Get #18828
    • IndexLookUpJoin #19721
    • IndexNestedLoopHashJoin #19721
    • HashJoinExec #19721
    • Load data statement #20713
    • HashAggExec, #20577
    • StreamAggExec, #20861
    • IndexLookUpExecutor, #19349
    • IndexMergeReaderExecutor, #20653
    • UnionScanExec, #20865
    • NestedLoopApplyExec
    • ParallelNestedLoopApplyExec
    • ProjectionExec
    • SplitIndexRegionExec
    • SplitTableRegionExec
    • ShowDDLJobQueriesExec
  • Add documents.
  • Add more tests.

Contributor guide

Research direction
Study the implementation of runtime stats in already supported executors (e.g., HashJoinExec) and apply similar approach to remaining executors like StreamAggExec, UnionScanExec, etc. Focus on collecting metrics such as execution time, rows processed, etc.
Tech stack
go
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Go
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30