apache/seatunnel

[Bug] [Connector-V2-File/HDFS] NoSuchMethodError for FsTracer.get with Hadoop 3.1.1 and Kerberos

Open

#10172 opened on Dec 10, 2025

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (6,897 stars) (1,432 forks)batch import
bughelp wanted

Description

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

Problem Description

When using SeaTunnel to write data from PostgreSQL to HDFS with Kerberos authentication enabled, the task fails to start and throws the exception java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get. The Kerberos authentication itself is successful, but a Hadoop API incompatibility error occurs when initializing the HDFS client.

Environment Information

  • SeaTunnel Version: 2.3.8
  • Running Mode: SeaTunnel Engine (Local)
  • Java Version: 1.8.0_291
  • Source Database: PostgreSQL
  • Target HDFS Version: Hadoop 3.1.1 (with Kerberos enabled)
  • Client Hadoop Environment: Not explicitly specified, suspected to have a version conflict with the built-in SeaTunnel version or the version from environment variables.

Task Configuration File (config/pg-to-hdfs.conf)

env {
  job.mode = "BATCH"
  execution.parallelism = 1
}

source {
  Jdbc {
    url = "jdbc:postgresql://157.3.xx.xx:5432/sqjw?gssEncMode=disable&sslmode=disable"
    driver = "org.postgresql.Driver"
    user = "root"
    password = "******"
    query = "SELECT * from tb limit 100"
    fetch_size = 1000
    result_table_name = "tb_data"
  }
}

sink {
  HdfsFile {
    path = "/db/hive/warehouse/YBSS/test-seatunnel"
    file_format_type = "orc"
    fs.defaultFS = "hdfs://hacluster"
    dfs.client.use.datanode.hostname = "true"
    hadoop.security.authentication = "kerberos"
    kerberos_principal = "Botest@HADOOP.COM"
    krb5_path = "/opt/krb5.conf"
    kerberos_keytab_path = "/opt/user.keytab"
# I also tried to configure using the hadoopConfig {} block, but the problem remains the same.  
 }
}

### SeaTunnel Version

2.3.8

### SeaTunnel Config

```conf
env {
  job.mode = "BATCH"
  execution.parallelism = 1
}

source {
  Jdbc {
    url = "jdbc:postgresql://157.3.xx.xx:5432/sqjw?gssEncMode=disable&sslmode=disable"
    driver = "org.postgresql.Driver"
    user = "root"
    password = "******"
    query = "SELECT * from tb limit 100"
    fetch_size = 1000
    result_table_name = "tb_data"
  }
}

sink {
  HdfsFile {
    path = "/db/hive/warehouse/YBSS/test-seatunnel"
    file_format_type = "orc"
    fs.defaultFS = "hdfs://hacluster"
    dfs.client.use.datanode.hostname = "true"
    hadoop.security.authentication = "kerberos"
    kerberos_principal = "Botest@HADOOP.COM"
    krb5_path = "/opt/krb5.conf"
    kerberos_keytab_path = "/opt/user.keytab"
# I also tried to configure using the hadoopConfig {} block, but the problem remains the same. 
 }
}

Running Command

bin/seatunnel.sh -c config/pg-to-hdfs.conf -m local

Error Exception

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/tracing/Tracer;
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:323)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:308)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initDFSClient(DistributedFileSystem.java:204)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:189)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.lambda$initializeWithKerberosLogin$0(HadoopFileSystemProxy.java:256)
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.lambda$loginWithKerberos$0(HadoopLoginFactory.java:52)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.loginWithKerberos(HadoopLoginFactory.java:50)
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initializeWithKerberosLogin(HadoopFileSystemProxy.java:249)
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initialize(HadoopFileSystemProxy.java:215)
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getFileSystem(HadoopFileSystemProxy.java:171)
	at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy.getOrCreateWriter(OrcWriteStrategy.java:124)
	at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy.write(OrcWriteStrategy.java:74)
	at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:134)
	at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:46)
	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:249)
	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:66)
	at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
	at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
	at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75)
	at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
	at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
	at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
	at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693)
	at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
	at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:205)

Zeta or Flink or Spark Version

zeta

Java or Scala Version

1.8.0_291

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
java
Domain
backenddata
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaHadoopKerberos authentication
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue is a NoSuchMethodError for org.apache.hadoop.fs.FsTracer.get, likely due to a version mismatch between the Hadoop client bundled with SeaTunnel and the target Hadoop 3.1.1 cluster. Investigate the Hadoop version used by SeaTunnel (particularly in HadoopFileSystemProxy.java and HadoopLoginFactory.java) and compare with the cluster's version. Check if FsTracer.get method exists in the classpath. Consider adding a dependency exclusion or shading the Hadoop client to avoid conflicts. Also verify Kerberos configuration is correctly propagating the authentication.