apache/seatunnel

[Bug] [seatunnel-engine-server] the slots will leak when the active master switches

Open

#9589 opened on Jul 17, 2025

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Java (6,897 stars) (1,432 forks)batch import
bughelp wanted

Description

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

We assume that there are 2 master nodes, m1 and m2, and the active master is m1. When m1 restarts, m2 will become the active master and restore all the running jobs. m2 will restore a running job as following steps:

  1. Change the job state from RUNNING to PENDING. CoordinatorService#restoreJobFromMasterActiveSwitch

  2. Apply slots. CoordinatorService#pendingJobSchedule

  3. Run the job. 3.1 CoordinatorService#pendingJobSchedule

3.2 PhysicalPlan#stateProcess We will execute the code in the red box because of the step 1 (job state is PENDING). But the code in the green box will return false because we only change the job state in step 1 and the pipeline state is still RUNNING instead of CREATED. This will result in the job not being deployed to the worker (i don't think it‘s necessary because the old job is still running on the worker) and the slots we applied in step 2 not being released.

After I print some logs it shows as below

SeaTunnel Version

2.3.11

SeaTunnel Config

seatunnel:
  engine:
    slot-service:
      dynamic-slot: false
      slot-num: 16

Running Command

//

Error Exception

This will result in NoEnoughResourceException finally.

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

JDK 1.8

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
java
Domain
backendinfrastructure
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Javadistributed systems basicsApache SeaTunnel engine knowledge
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Investigate how the pipeline state is handled during master switch in CoordinatorService. The bug occurs in restoreJobFromMasterActiveSwitch when job state is changed to PENDING but pipeline state remains RUNNING. Examine PhysicalPlan#stateProcess and PipelineState to ensure pipeline state is also reset to CREATED. Also consider releasing slots if deployment fails in pendingJobSchedule.