[CELEBORN] FailedShuffleCleaner instance is never returned to CelebornShuffleManager, leaking its ScheduledExecutorService on stop()
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- distributed-systems
Research direction
Start in CelebornUtils, where fetchCleanFailedShuffle and stageRerun take failedShuffleCleaner as a parameter and assign a new FailedShuffleCleaner to it, then check the call sites in CelebornShuffleManager, which holds the failedShuffleCleaner field and its stop() method. Done means the created cleaner is returned to the caller, stored in the manager's field, and shut down in stop(), with a test that confirms the field is non-null after a stage rerun and that the executor is released on stop.
Written by the indexing model from the issue text.
Description
Backend
VL (Velox)
Bug description
CelebornUtils.fetchCleanFailedShuffle(lifecycleManager, celebornConf, failedShuffleCleaner) and stageRerun(...) take failedShuffleCleaner as an output parameter and assign the newly created FailedShuffleCleaner instance to it.
Because Java is pass-by-value, reassigning a method parameter is not visible to the caller — CelebornShuffleManager.failedShuffleCleaner stays null forever.
org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
becomes dead code. The cleaner's internal ScheduledExecutorService is never shut down, and keeps strong references to LifecycleManager and its pending shuffle-id collections, leaking both threads and objects. This accumulates across SparkContext lifecycles and is most visible on long-lived drivers (e.g., Spark Thrrift Server), where each restart leaves orphaned cleaner threads and repeated invalid clean log entries.
Expected behavior
stageRerun/fetchCleanFailedShuffle should return the created instance so the caller can store it and stop() can release it.
Gluten version
No response
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day