apache/seatunnel
View on GitHub[Bug][S3] Unable to access S3 using role and webIdentityTokenFile in WebIdentityTokenFileCredentialsProvider
Open
#9,233 opened on Apr 25, 2025
help wanted
Description
Search before asking
- #9240
Description
现状: 目前seatunnel 服务部署在K8s 中, 需要将Mysql 数据通过 mysql cdc同步到 Hive(hive on s3)中, 看起来目前只提供了as/sk 方式进行S3访问,能否功能使 POD的中 webIdentityTokenFile 和 role 访问S3?
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 10000
}
source {
MySQL-CDC {
plugin_output = "fake"
base-url = "jdbc:mysql://test:3306/test"
username = "***"
password = "****"
table-names = ["test.user"]
table-names-config = [
{
table = "test.user"
primaryKeys = ["id"]
}
]
}
}
transform {
Sql {
plugin_input = "fake"
plugin_output = "fake1"
query = "select id, open_id from user"
}
}
sink {
Hive {
plugin_input = "fake1"
table_name = "tmp.test_hive_sink_on_s3"
metastore_uri = "thrift://ip-xxx-xx-1-142.cn-north-1.compute.internal:9083"
hive.hadoop.conf = {
bucket="s3a://xxxxxxx"
fs.s3a.endpoint="s3.cn-north-1.amazonaws.com.cn"
fs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.WebIdentityTokenFileCredentialsProvider" //期望
}
}
}
参考: https://mvnrepository.com/artifact/software.amazon.awssdk/auth/2.31.28
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct