[CELEBORN] FailedShuffleCleaner instance is never returned to CelebornShuffleManager, leaking its ScheduledExecutorService on stop()

Open
#13,279 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java, scala, spark

Research direction

Start in CelebornUtils, where fetchCleanFailedShuffle and stageRerun take failedShuffleCleaner as a parameter and assign a new FailedShuffleCleaner to it, then check the call sites in CelebornShuffleManager, which holds the failedShuffleCleaner field and its stop() method. Done means the created cleaner is returned to the caller, stored in the manager's field, and shut down in stop(), with a test that confirms the field is non-null after a stage rerun and that the executor is released on stop.

Written by the indexing model from the issue text.

Description

bug triage
Backend

VL (Velox)

Bug description

CelebornUtils.fetchCleanFailedShuffle(lifecycleManager, celebornConf, failedShuffleCleaner) and stageRerun(...) take failedShuffleCleaner as an output parameter and assign the newly created FailedShuffleCleaner instance to it.

Because Java is pass-by-value, reassigning a method parameter is not visible to the caller — CelebornShuffleManager.failedShuffleCleaner stays null forever.

org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager

Image Image

becomes dead code. The cleaner's internal ScheduledExecutorService is never shut down, and keeps strong references to LifecycleManager and its pending shuffle-id collections, leaking both threads and objects. This accumulates across SparkContext lifecycles and is most visible on long-lived drivers (e.g., Spark Thrrift Server), where each restart leaves orphaned cleaner threads and repeated invalid clean log entries.

Expected behavior
stageRerun/fetchCleanFailedShuffle should return the created instance so the caller can store it and stop() can release it.

Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.