pingcap/tiflow

[TiCDC] support data compression when syncing to MySQL/TiDB

Open

#9,733 opened on Sep 13, 2023

View on GitHubContributor guide
 (3 comments) (1 reaction) (0 assignees)Go (310 forks)auto 404
help wantedtype/feature

Repository metrics

Stars
 (461 stars)
PR merge metrics
 (PR metrics pending)

Description

Is your feature request related to a problem?

When deploy multiple TiDB Clusters across multiple regions, TiCDC will be used to replicate data changes between these TiDB Clusters. There are some points need to be considered when doing cross region replication:

  • The network bandwidth of cross regions is limited.
  • The cost of data transfer across regions is higher than data transfer inside a region.

Describe the feature you'd like

Support compression when replicate data changes across regions. MySQL protocol support compression: https://dev.mysql.com/doc/refman/8.0/en/connection-compression-control.html TiDB also support connection compression: https://github.com/pingcap/tidb/issues/22605

Describe alternatives you've considered

  1. Don't support compression, pay more cost for cross region replication; but it is easier to hit cross region network bandwidth limitation when the data replication throughput is large enough.

Teachability, Documentation, Adoption, Migration Strategy

No response

Contributor guide

Research direction
Study MySQL compression protocol and TiDB's existing compression implementation (PR #22605). Then implement buffered write with compression in the TiCDC sink module for MySQL/TiDB targets.
Tech stack
go
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GoMySQLTiDB
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50