apache/seatunnel

[Umbrella] unified parameter class and refactor connector creation logic

Closed

#8576 opened on Jan 23, 2025

View on GitHub
 (30 comments) (3 reactions) (0 assignees)Java (6,897 stars) (1,432 forks)batch import
good first issuehelp wanted

Description

Code of Conduct

Search before asking

  • I had searched in the issues and found no similar issues.

Describe the proposal

This issue has two proposal:

  1. move all parameter that user can adjust to a single class, connector use XXXSourceConfig/ XXXSourceOptions. Update EnvCommandOptions to describe the parameters we can adjust in config env part. Update ServerConfigOptions to describe the parameters we can adjust about Zeta Engine config (config in SeaTunnel.yaml).

The benefits:

  • We can easily know what parameter can be adjusted, that can help us to tune the performance.
  • When version release, we can easily know which parameter has been changed, which can break the user version upgrade, which is added in new version, So we can record this into version upgrade document (although we don't have it yet).
  1. refactor connector, implement TableSourceFactory, TableSinkFactory interface and implement createSource, createSink, optionRule method.

In optionRule method, describe which parameter is necessary, which is optional, which are the conflicts with other parameter. (need action1 done)

Move the connector creation logic from Connector to ConnectorFactory.

The benefits:

  • Know the relationship between parameters. Use the optionRule check the config before create connector instance.
  • This can be used in SeaTunnelWeb.

Welcome to do this with me

Task list

type assign to status
ZetaConfig ServerConfigOptions @liunaijie Done
EnvConfig EnvCommonOptions @liunaijie Done
Connector connector-activemq @liunaijie Done
Connector connector-amazondynamodb @liunaijie Done
Connector connector-amazonsqs @liunaijie Done
Connector connector-assert @liunaijie Done
Connector connector-cassandra @liunaijie Done
Connector connector-clickhouse @lyne7-sc Done
Connector connector-console @liunaijie Done
Connector connector-datahub @liunaijie Done
Connector connector-dingtalk @liunaijie Done
Connector connector-doris @liunaijie Done
Connector connector-druid @akulabs8 Done
Connector connector-easysearch @liunaijie Done
Connector connector-elasticsearch @fcb-xiaobo Done
Connector connector-email @xdu-chenrj Done
Connector connector-fake @liunaijie Done
Connector connector-file-cos @misi1987107 Done
Connector connector-file-ftp @misi1987107 Done
Connector connector-file-hadoop @misi1987107 Done
Connector connector-file-jindo-oss @misi1987107 Done
Connector connector-file-local @misi1987107 Done
Connector connector-file-obs @misi1987107 Done
Connector connector-file-oss @misi1987107 Done
Connector connector-file-s3 @litiliu Done
Connector connector-file-sftp @misi1987107 Done
Connector connector-google-firestore @liunaijie Done
Connector connector-google-sheets @liunaijie Done
Connector connector-hbase @liunaijie Done
Connector connector-http-feishu @liunaijie Done
Connector connector-http-github @liunaijie Done
Connector connector-http-gitlab @liunaijie Done
Connector connector-http-jira @liunaijie Done
Connector connector-http-klaviyo @liunaijie Done
Connector connector-http-lemlist @liunaijie Done
Connector connector-http-myhours @liunaijie Done
Connector connector-http-notion @liunaijie Done
Connector connector-http-onesignal @liunaijie Done
Connector connector-http-persistiq @liunaijie Done
Connector connector-http-wechat @liunaijie Done
Connector connector-hudi @liunaijie Done
Connector connector-iceberg @liunaijie Done
Connector connector-influxdb @liunaijie Done
Connector connector-iotdb @liunaijie Done
Connector connector-jdbc @liunaijie Done
Connector connector-kafka @wattt3 Done
Connector connector-kudu @liunaijie Done
Connector connector-maxcompute @liunaijie Done
Connector connector-milvus @liunaijie Done
Connector connector-mongodb @liunaijie Done
Connector connector-neo4j @liunaijie Done
Connector connector-openmldb @liunaijie Done
Connector connector-paimon @liunaijie Done
Connector connector-prometheus @liunaijie Done
Connector connector-pulsar @liunaijie Done
Connector connector-qdrant @liunaijie Done
Connector connector-rabbitmq @liunaijie Done
Connector connector-redis @akulabs8 Done
Connector connector-rocketmq @liunaijie Done
Connector connector-selectdb-cloud @liunaijie Done
Connector connector-sentry @liunaijie Done
Connector connector-slack @akulabs8 Done
Connector connector-sls @liunaijie Done
Connector connector-socket @liunaijie Done
Connector connector-starrocks @fcb-xiaobo Done
Connector connector-tablestore @liunaijie Done
Connector connector-tdengine @liunaijie Done
Connector connector-typesense @liunaijie Done
Connector connector-web3j @akulabs8 Done
Connector connector-cdc-mongodb @liunaijie Done
Connector connector-cdc-mysql @liunaijie Done
Connector connector-cdc-opengauss @liunaijie Done
Connector connector-cdc-oracle @liunaijie Done
Connector connector-cdc-postgres @liunaijie Done
Connector connector-cdc-sqlserver @liunaijie Done
Connector connector-cdc-tidb @liunaijie Done
Connector connector-s3-redshift @liunaijie Done
Connector connector-hive @liunaijie Done

What should a Pull Request accomplish?

  • List and describe the connector parameters in the Factory.optionRule() method. While doing so, we can move all parameters into the XXXXOptions class to unify the code style and centralize the parameters into a single file.

  • Move the connector creation logic into Factory.createSource/createSink. Implement the createSource/createSink methods in Factory and instantiate the corresponding connector.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Contributor guide