[Bug][S3] Unable to access S3 using role and webIdentityTokenFile in WebIdentityTokenFileCredentialsProvider · apache/seatunnel#9233

(5 commenti) (0 reazioni) (0 assegnatari)Java (1432 fork)batch import

help wanted

Metriche repository

Star: (6897 star)
Metriche merge PR: (Merge medio 13g 21h) (143 PR mergiate in 30 g)

Descrizione

Search before asking

#9240

Description

现状：目前seatunnel 服务部署在K8s 中，需要将Mysql 数据通过 mysql cdc同步到 Hive（hive on s3）中，看起来目前只提供了as/sk 方式进行S3访问，能否功能使 POD的中 webIdentityTokenFile 和 role 访问S3？

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 10000
}

source {
  MySQL-CDC {
    plugin_output = "fake"
    base-url = "jdbc:mysql://test:3306/test"
    username = "***"
    password = "****"
    
    table-names = ["test.user"]
    table-names-config = [
      {
        table = "test.user"
        primaryKeys = ["id"]
      }
    ]
  }
}

transform {
  Sql {
    plugin_input = "fake"
    plugin_output = "fake1"
    query = "select id, open_id from user"
  }
}

sink {
  Hive {
    plugin_input = "fake1"
    table_name = "tmp.test_hive_sink_on_s3"
    metastore_uri = "thrift://ip-xxx-xx-1-142.cn-north-1.compute.internal:9083"
    hive.hadoop.conf = {
        bucket="s3a://xxxxxxx"
        fs.s3a.endpoint="s3.cn-north-1.amazonaws.com.cn"
        fs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.WebIdentityTokenFileCredentialsProvider" //期望
    }
  }
}

参考： https://mvnrepository.com/artifact/software.amazon.awssdk/auth/2.31.28 software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider

Usage Scenario

No response

Related issues