apache/seatunnel

[Bug] [Module Name] TiDB-CDC cdc event lose data

Open

#8.815 geöffnet am 24. Feb. 2025

Auf GitHub ansehen
 (8 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Java (1.432 Forks)batch import
bughelp wanted

Repository-Metriken

Stars
 (6.897 Stars)
PR-Merge-Metriken
 (Durchschn. Merge 13T 21h) (143 gemergte PRs in 30 T)

Beschreibung

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

Data is lost during the cdc phase,not all table is missing, and a few table will appear。

regionId: 12819225739 startKey: t_656992_5f72800000000000eadd endKey: t_657156_5f698000000000000001038000000008f76ec203814512fa84f0092c038000000000370de0

SeaTunnel Version

2.3.8

SeaTunnel Config

{
    "env": {
        "parallelism": 1,
        "job.mode": "STREAMING",
        "job.name": "tidb=>Kafka : cxxx",
        "checkpoint.interval": "60000",
        "checkpoint.timeout": "6000",
        "flush.timeout.ms": "6000"
    },
    "source" : [
        {
            "plugin_name": "TiDB-CDC-MIGRATE",
            "pd-addresses": "xxx:2379",
            "driver": "com.mysql.cj.jdbc.Driver",
            "base-url" : "jdbc:mysql://xxx?useSSL=false",
            "database-name" : "xxx",
            "username": "xxx",
            "password": "xxx",
            "table-name": "xxx",
            "tikv.grpc.timeout_in_ms": "260000"
        }
    ],
    "transform" : [
        
    ],
    "sink" : [
        {
            "plugin_name" : "Kafka",
            "bootstrap.servers" : "xxx:xx",
            "topic": "short_video_log_ext.epis_watch_log",
	"kafka.config": {
		"acks": "all",
                "retries": 128,
                "retry.backoff.ms": 500,
		"request.timeout.ms": 240000,
                "batch.size": 10240,
                "buffer.memory": 67108864,
                "send.buffer.bytes": 262144,
                "compression.gzip.level": 6,
                "compression.type": "gzip"
            },
            "semantics" : "AT_LEAST_ONCE",
            "format": "debezium_json"
        }
    ]
}

Running Command

web http commit

Error Exception

no error message

Zeta or Flink or Spark Version

zeta

Java or Scala Version

jdk21

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor Guide