br: open format file generation by backup · pingcap/tidb#58611

(0 留言) (0 反應) (0 負責人)Go (40,090 star) (6,186 fork)batch import

help wantedtype/feature-request

描述

Feature Request

Is your feature request related to a problem? Please describe:

Currently, there is no native tidb tool to export tikv data into open format files, like parquet. Instead, tidb users need to use client like tispark etc to extract data and do the format conversion. And the long tech stack suffers from bad performance.

Describe the feature you'd like:

tidb can provide a native way to dump snapshot data and incremental data to open format files. A preferred way is to let backup to generate open format files directly, in other words, backup can support to generate either log/sst files or parquet files. A simple prototype code is here https://github.com/BornChanger/sampleParquet.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

貢獻者指南

技術棧: go
領域: databasedatabackend
議題類型: feature
難度: 4
預計時間: over 1 week
活動狀態: fresh
清晰度: clear
前置要求: GolangTiDB backup internalsParquet file format
新手友善度: 20
研究方向: The issue requests adding native Parquet export to TiDB backup. The prototype code at https://github.com/BornChanger/sampleParquet provides a starting point. The implementation will likely involve modifying the backup module (e.g., `br` in the TiDB repository) to write Parquet files instead of SST or log files. Look into how backup currently writes data and explore the Parquet Go library (e.g., apache/arrow go) for integration. The feature requires understanding of both TiDB's snapshot/incremental backup mechanisms and Parquet schema generation.