apache/seatunnel

[Feature][Connector-v2][MySQL] Support dynamic discovery of new tables for regex-based multi-table sync

Open

#10203 opened on Dec 17, 2025

View on GitHub
 (18 comments) (0 reactions) (1 assignee)Java (6,897 stars) (1,432 forks)batch import
good first issuehelp wanted

Description

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

Search before asking

  • I have searched the issues and found no similar feature request.

Description

When using the MySQL connector to synchronize multiple tables, we can configure a regex pattern to match table names (e.g. test_.*) so that multiple tables are synchronized in one job.

However, currently the job only discovers and synchronizes the tables that exist at job startup. If new tables are created in the same database later and their names also match the configured regex pattern, these newly created tables are not automatically picked up and synchronized by the running job.

Expected Behavior

I would like SeaTunnel to support dynamic discovery of new tables for MySQL regex-based multi-table sync:

  • A job is configured with a regex pattern for tables (for example test_.*).
  • After the job has started, when a new table (e.g. test_3) is created in the same database and its name matches the configured pattern, the connector should:
    • Automatically discover this new table, and
    • Start reading/synchronizing data from this new table without restarting the job.

This could be controlled by a new configuration option (for example discover-new-tables / dynamic_table_discovery), so users can enable or disable this behavior explicitly.

Why is this needed?

For scenarios where new tables are created frequently following the same naming convention (for example, sharded tables or time-partitioned tables), it is very inconvenient to modify the configuration and restart the job every time a new table is added.

Dynamic table discovery would:

  • Reduce operational overhead.
  • Make SeaTunnel more suitable for large-scale, frequently-changing MySQL schemas.
  • Align with the expectation that a regex pattern should keep matching newly created tables as long as they satisfy the pattern.

Additional context

  • Connector: MySQL (CDC / source used for syncing multiple tables with regex).
  • Behavior observed: only tables existing at job start are synchronized; newly created matching tables are ignored until the job is updated/restarted.

Please let me know if there is already a way to enable this behavior in the current version, or if more information is needed.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
javamysqlsql
Domain
backenddatadatabase
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
JavaMySQL connectorSeaTunnel source connector architectureCDC concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Investigate the current table discovery logic in the MySQL source connector, likely in classes such as MySQLSource or TableDiscoveryUtils. Understand how the regex pattern is applied at startup. Then design a periodic polling mechanism to check for new tables matching the pattern. Key decisions include polling interval and handling of data consistency. Review existing issue comments and any related PRs for similar patterns in other connectors.