apache/seatunnel

The submitted tasks remain in PENDING status

Open

#10.993 aberto em 2 de jun. de 2026

Ver no GitHub
 (0 comments) (0 reactions) (0 assignees)Java (1.432 forks)batch import
bughelp wanted

Métricas do repositório

Stars
 (6.897 stars)
Métricas de merge de PR
 (Mesclagem média 25d 9h) (90 fundiu PRs em 30d)

Description

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

The SeaTunnel cluster is deployed in cluster mode with 1 master node and 1 worker node. All offline tasks are running on the platform with a minute-level scheduling cycle. Currently, 160 tasks are executing, and subsequently submitted tasks fall into pending status. Relevant logs are as follows:

Log.txt

SeaTunnel Version

v2.3.11

SeaTunnel Config

master configuration：jvm_master_options
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# JVM Heap
-Xms30g
-Xmx30g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=2g

# G1GC
-XX:+UseG1GC


Configuration of seatunnel.yaml：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

seatunnel:
  engine:
    classloader-cache-mode: true
    history-job-expire-minutes: 90
    backup-count: 0
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: false
      slot-num: 10
    job-schedule-strategy: WAIT
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          namespace: /tmp/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission
    telemetry:
      metric:
        enabled: false
      logs:
        scheduled-deletion-enable: true
    http:
      enable-http: true
      port: 18080
      enable-dynamic-port: false
      # Uncomment the following lines to enable basic authentication for web UI
      # enable-basic-auth: true
      # basic-auth-username: admin
      # basic-auth-password: admin
    event-report-http:
      #url: "http://172.23.74.43:9000/event/report"
      url: "http://172.23.57.233:9000/event/report"
      headers:
        Content-Type: application/json

coordinator-service:
  core-thread-num: 30
  max-thread-num: 40


Worker configuration：
jvm_worker_options：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# JVM Heap
-Xms20g
-Xmx20g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=2g

# G1GC
-XX:+UseG1GC


seatunnel.yaml：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

seatunnel:
  engine:
    classloader-cache-mode: true
    history-job-expire-minutes: 20
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: false
      slot-num: 20
    job-schedule-strategy: WAIT
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          namespace: /tmp/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission
    telemetry:
      metric:
        enabled: false
      logs:
        scheduled-deletion-enable: true
    http:
      enable-http: true
      port: 8080
      enable-dynamic-port: false
      # Uncomment the following lines to enable basic authentication for web UI
      # enable-basic-auth: true
      # basic-auth-username: admin
      # basic-auth-password: admin

Running Command

/home/seatunnel/apache-seatunnel-2.3.11/bin/seatunnel.sh -c /home/seatunnel/seatunnel_test/test1.json

Error Exception

2026-06-02 11:37:56,345 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (1114034841512312835) end with state FAILED
2026-06-02 11:37:56,346 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2026-06-02 11:37:56,350 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node03]:5802:0a3c2c7b-1c80-4fa9-a879-75ffa9636d88, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/172.23.57.231:36645->node03/172.23.57.233:5802}, remoteAddress=[node03]:5802, lastReadTime=2026-06-02 11:37:48.572, lastWriteTime=2026-06-02 11:37:48.571, closedTime=2026-06-02 11:37:56.347, connected server version=5.1}
2026-06-02 11:37:56,351 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node02]:5802:c5b34c3d-7f0d-4319-9ab7-a61c42d136b8, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/172.23.57.231:41599->node02/172.23.57.232:5802}, remoteAddress=[node02]:5802, lastReadTime=2026-06-02 11:37:53.489, lastWriteTime=2026-06-02 11:37:53.489, closedTime=2026-06-02 11:37:56.350, connected server version=5.1}
2026-06-02 11:37:56,352 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node01]:5801:84805bed-fd34-47c2-8819-b827d613008a, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.23.57.231:46309->node01/172.23.57.231:5801}, remoteAddress=[node01]:5801, lastReadTime=2026-06-02 11:37:56.337, lastWriteTime=2026-06-02 11:37:53.572, closedTime=2026-06-02 11:37:56.351, connected server version=5.1}
2026-06-02 11:37:56,352 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2026-06-02 11:37:56,356 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2026-06-02 11:37:56,356 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2026-06-02 11:37:56,356 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2026-06-02 11:37:56,356 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 

===============================================================================


2026-06-02 11:37:56,356 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error, 

2026-06-02 11:37:56,356 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2026-06-02 11:37:56,357 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed 

2026-06-02 11:37:56,361 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:282)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:278)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:397)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:182)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48)
        at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:42)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000ms while awaiting InitProducerId

        ... 12 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
        ... 2 more
 
2026-06-02 11:37:56,361 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 
===============================================================================



Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:282)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:278)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:397)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:182)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48)
        at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:42)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000ms while awaiting InitProducerId

        ... 12 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
        ... 2 more
2026-06-02 11:37:56,364 INFO  [s.c.s.s.c.ClientExecuteCommand] [SeaTunnel-CompletableFuture-Thread-0] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Guia do colaborador

Direção de pesquisa
Examine o errore do CheckpointCoordinator e o timeout do Kafka InitProducerId. Rastreie o caminho do código em CheckpointCoordinator.handleCoordinatorError e CheckpointErrorReportOperation para entender por que as tarefas ficam no estado PENDING.
Pilha de tecnologia
javakafka
Domain
backend
Tipo Issue
Bug
DifficultyDificuldade de implementação estimada para um novo contribuidor, de 1 para alterações muito pequenas a 5 para trabalho de nível especializado.
2
Tempo estimadoUm intervalo de tempo aproximado para um colaborador experiente investigar, implementar, testar e preparar um pull request.
1-3 horas
Status da atividadeQuão disponível o issue aparece agora: novo, ativo, obsoleto, bloqueado ou aguardando entrada do mantenedor.
Ativo
ClarityCom que clareza o issue explica a mudança esperada, os critérios de aceitação e a próxima etapa.
Principalmente claro
Prerequisites
Java
Simpatia para novatosUma pontuação de 1 a 100 que estima o quão acessível este issue é para colaboradores iniciantes.
50