The submitted tasks remain in PENDING status · apache/seatunnel#10993

(0 commenti) (0 reazioni) (0 assegnatari)Java (6897 star) (1432 fork)batch import

bughelp wanted

Descrizione

Search before asking

I had searched in the issues and found no similar issues.

What happened

The SeaTunnel cluster is deployed in cluster mode with 1 master node and 1 worker node. All offline tasks are running on the platform with a minute-level scheduling cycle. Currently, 160 tasks are executing, and subsequently submitted tasks fall into pending status. Relevant logs are as follows:

Log.txt

SeaTunnel Version

v2.3.11

SeaTunnel Config

master configuration：jvm_master_options
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# JVM Heap
-Xms30g
-Xmx30g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=2g

# G1GC
-XX:+UseG1GC


Configuration of seatunnel.yaml：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

seatunnel:
  engine:
    classloader-cache-mode: true
    history-job-expire-minutes: 90
    backup-count: 0
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: false
      slot-num: 10
    job-schedule-strategy: WAIT
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          namespace: /tmp/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission
    telemetry:
      metric:
        enabled: false
      logs:
        scheduled-deletion-enable: true
    http:
      enable-http: true
      port: 18080
      enable-dynamic-port: false
      # Uncomment the following lines to enable basic authentication for web UI
      # enable-basic-auth: true
      # basic-auth-username: admin
      # basic-auth-password: admin
    event-report-http:
      #url: "http://172.23.74.43:9000/event/report"
      url: "http://172.23.57.233:9000/event/report"
      headers:
        Content-Type: application/json

coordinator-service:
  core-thread-num: 30
  max-thread-num: 40


Worker configuration：
jvm_worker_options：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# JVM Heap
-Xms20g
-Xmx20g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=2g

# G1GC
-XX:+UseG1GC


seatunnel.yaml：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

seatunnel:
  engine:
    classloader-cache-mode: true
    history-job-expire-minutes: 20
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: false
      slot-num: 20
    job-schedule-strategy: WAIT
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          namespace: /tmp/seatunnel/checkpoint_snapshot
          storage.type: hdfs
          fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission
    telemetry:
      metric:
        enabled: false
      logs:
        scheduled-deletion-enable: true
    http:
      enable-http: true
      port: 8080
      enable-dynamic-port: false
      # Uncomment the following lines to enable basic authentication for web UI
      # enable-basic-auth: true
      # basic-auth-username: admin
      # basic-auth-password: admin

Running Command

/home/seatunnel/apache-seatunnel-2.3.11/bin/seatunnel.sh -c /home/seatunnel/seatunnel_test/test1.json

Error Exception

2026-06-02 11:37:56,345 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (1114034841512312835) end with state FAILED
2026-06-02 11:37:56,346 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2026-06-02 11:37:56,350 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node03]:5802:0a3c2c7b-1c80-4fa9-a879-75ffa9636d88, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/172.23.57.231:36645->node03/172.23.57.233:5802}, remoteAddress=[node03]:5802, lastReadTime=2026-06-02 11:37:48.572, lastWriteTime=2026-06-02 11:37:48.571, closedTime=2026-06-02 11:37:56.347, connected server version=5.1}
2026-06-02 11:37:56,351 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node02]:5802:c5b34c3d-7f0d-4319-9ab7-a61c42d136b8, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/172.23.57.231:41599->node02/172.23.57.232:5802}, remoteAddress=[node02]:5802, lastReadTime=2026-06-02 11:37:53.489, lastWriteTime=2026-06-02 11:37:53.489, closedTime=2026-06-02 11:37:56.350, connected server version=5.1}
2026-06-02 11:37:56,352 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [node01]:5801:84805bed-fd34-47c2-8819-b827d613008a, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.23.57.231:46309->node01/172.23.57.231:5801}, remoteAddress=[node01]:5801, lastReadTime=2026-06-02 11:37:56.337, lastWriteTime=2026-06-02 11:37:53.572, closedTime=2026-06-02 11:37:56.351, connected server version=5.1}
2026-06-02 11:37:56,352 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2026-06-02 11:37:56,356 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2026-06-02 11:37:56,356 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2026-06-02 11:37:56,356 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2026-06-02 11:37:56,356 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 

===============================================================================


2026-06-02 11:37:56,356 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error, 

2026-06-02 11:37:56,356 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2026-06-02 11:37:56,357 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed 

2026-06-02 11:37:56,361 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:282)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:278)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:397)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:182)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48)
        at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:42)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000ms while awaiting InitProducerId

        ... 12 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
        ... 2 more
 
2026-06-02 11:37:56,361 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 
===============================================================================



Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:282)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:278)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:397)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:182)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.runInternal(CheckpointErrorReportOperation.java:48)
        at org.apache.seatunnel.engine.server.task.operation.TracingOperation.run(TracingOperation.java:42)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:471)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:197)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:137)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000ms while awaiting InitProducerId

        ... 12 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:220)
        ... 2 more
2026-06-02 11:37:56,364 INFO  [s.c.s.s.c.ClientExecuteCommand] [SeaTunnel-CompletableFuture-Thread-0] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Guida contributor

Tech stack: javakafka
Dominio: backend
Tipo issue: bug
Difficoltà: 2
Tempo stimato: 1-3 hours
Stato attività: active
Chiarezza: mostly clear
Prerequisiti: Java
Adatta ai principianti: 50
Direzione di ricerca: Esamina l'errore CheckpointCoordinator e il timeout di Kafka InitProducerId. Traccia il percorso del codice in CheckpointCoordinator.handleCoordinatorError e CheckpointErrorReportOperation per capire perché le attività rimangono in stato PENDING.