[CELEBORN] FailedShuffleCleaner instance is never returned to CelebornShuffleManager, leaking its ScheduledExecutorService on stop()

Abierto
#13,279 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
Medio día
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
java, scala, spark

Línea de trabajo

Start in CelebornUtils, where fetchCleanFailedShuffle and stageRerun take failedShuffleCleaner as a parameter and assign a new FailedShuffleCleaner to it, then check the call sites in CelebornShuffleManager, which holds the failedShuffleCleaner field and its stop() method. Done means the created cleaner is returned to the caller, stored in the manager's field, and shut down in stop(), with a test that confirms the field is non-null after a stage rerun and that the executor is released on stop.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug triage
Backend

VL (Velox)

Bug description

CelebornUtils.fetchCleanFailedShuffle(lifecycleManager, celebornConf, failedShuffleCleaner) and stageRerun(...) take failedShuffleCleaner as an output parameter and assign the newly created FailedShuffleCleaner instance to it.

Because Java is pass-by-value, reassigning a method parameter is not visible to the caller — CelebornShuffleManager.failedShuffleCleaner stays null forever.

org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager

Image Image

becomes dead code. The cleaner's internal ScheduledExecutorService is never shut down, and keeps strong references to LifecycleManager and its pending shuffle-id collections, leaking both threads and objects. This accumulates across SparkContext lifecycles and is most visible on long-lived drivers (e.g., Spark Thrrift Server), where each restart leaves orphaned cleaner threads and repeated invalid clean log entries.

Expected behavior
stageRerun/fetchCleanFailedShuffle should return the created instance so the caller can store it and stop() can release it.

Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

Lenguaje dominante
Scala
Estrellas
1.6k
Forks
685
Merge medio
4 d 16 h
PR fusionados (30 d)
84

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/gluten

Todos los issues de apache/gluten

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.