apache/seatunnel

Update AWS SDK to support IRSA (IAM Roles for Service Accounts) in EKS

Open

#10302 opened on Jan 8, 2026

View on GitHub
 (3 comments) (0 reactions) (1 assignee)Java (6,897 stars) (1,432 forks)batch import
help wanted

Description

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

Description

SeaTunnel 2.3.12 ships with AWS SDK 1.11.271 (from March 2018), which does not support IRSA (IAM Roles for Service Accounts) authentication in Amazon EKS. This prevents using S3 checkpoint storage with modern EKS security patterns.

Current Behavior

When configuring S3 checkpoint storage in EKS with IRSA:

checkpoint:
  storage:
    type: hdfs
    plugin-config:
      storage.type: s3
      s3.bucket: s3a://my-bucket
      fs.s3a.endpoint: s3.amazonaws.com

Error:
com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider
InstanceProfileCredentialsProvider

Root Cause

1. AWS SDK 1.11.271 (released March 2018) predates IRSA support
2. IRSA was added to AWS SDK in version 1.11.704 (January 2020)
3. IRSA requires WebIdentityTokenCredentialsProvider which is not available in 1.11.271
4. Both aws-java-sdk-bundle-1.11.271.jar and seatunnel-hadoop-aws.jar contain the old AWS SDK

Expected Behavior

SeaTunnel should support IRSA authentication in EKS environments by:
- Automatically detecting AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables
- Using STSAssumeRoleWithWebIdentitySessionCredentialsProvider for authentication

Proposed Solution

Update AWS SDK version to 1.11.1034 or newer in seatunnel-dist/pom.xml:

<aws-java-sdk.version>1.11.1034</aws-java-sdk.version>

This was already attempted in PR #4468 (now closed), which successfully updated the AWS SDK but was closed due to missing E2E tests.

Environment

- SeaTunnel Version: 2.3.12
- Deployment: Amazon EKS (Kubernetes)
- Authentication: IRSA (IAM Roles for Service Accounts)
- Storage: S3 for checkpoint storage
- Current AWS SDK: 1.11.271 (March 2018)
- Required AWS SDK: 1.11.704+ (for IRSA support)

Related

- Closed PR #4468: Update AWS SDK to support IRSA
- IRSA was introduced in AWS SDK 1.11.704 (January 2020)



### Usage Scenario

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Tech stack
java
Domain
backendcloud
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaMaven
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Investigate the current AWS SDK version in seatunnel dist/pom.xml and update the `<aws java sdk.version>` to 1.11.1034 or newer. Review the closed PR #4468 for the previous implementation and ensure WebIdentityTokenCredentialsProvider is properly configured. Add E2E tests to verify IRSA authentication in an EKS environment, as the previous PR was closed due to missing tests. Consider testing with mock environment variables to avoid actual AWS costs.