Helix Controller Does Not Rebalance Partitions After A Participant Loss

Abierto
  • ¿Algo no está bien?
#3,049 2 comentarios 2 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
aws, docker, java
Área
cloud, distributed-systems, infrastructure

Línea de trabajo

Comienza siguiendo el manejo del rebalanceo FULL_AUTO de Apache Helix para el modelo de estado OnlineOffline durante una terminación forzada de AWS ECS Spot. Compara LIVEINSTANCES y EXTERNALVIEW después de que desaparezca el nodo, y luego compara el comportamiento del controlador con el reinicio manual del servicio que restaura las asignaciones. Se considera completado cuando las particiones que anteriormente pertenecían al nodo detenido se reasignan automáticamente a nodos activos sin reiniciar todos los servicios.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Problem

We are using Apache Helix to distribute a resource across multiple services - all registered under the same Helix cluster name and The system is deployed on AWS ECS Fargate using Docker containers.

Each service uses the OnlineOffline state model, with 1 replica per partition and the FULL_AUTO rebalance mode.

When all services start, Helix assigns partitions as expected and all services consume their assigned partitions. The problem occurs when a service is restarted - due to an AWS Spot Termination which appear to forcefully kill the container, bypassing the logic we have to handle SIGTERM

So when a service is restarted from the cluster, the partition previously assigned to the now-stopped node are not reassigned to new live nodes. As a result, that partition remains Idle, even though other nodes are available and registered in live instances.

I have tried with both Apache Helix version 0.9.9 and the latest 1.4.3, but the problem still persists in both versions. Also When a node is removed from the cluster, there are no logs from the controller instance indicating that it is attempting to rebalance the partitions.

Additional Context After ZooKeeper Inspection
  • /LIVEINSTANCES is perfectly listing only actively running nodes.
  • But in /EXTERNALVIEW for that partition still shows the stopped node as ONLINE and does not reassign it to a live node.
  • This mismatch persists even after new nodes are registered as live instances.
Expected behavior

Helix should detect that the node is no longer live and automatically reassign its partitions to other available nodes in the cluster.

What worked for us

But I have been able to temporarily resolve the issue by performing a manual restart of all services register under that HELIX CLUSTER. After the manual restart, all partitions are perfectly reassigned.

I will be active here and will respond to all your replies as soon as possible and I am happy to provide any additional information or logs that would help with debugging this issue.

Lenguaje dominante
Java
Estrellas
505
Forks
258
Merge medio
4 d 20 h
PR fusionados (30 d)
3

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/helix

Todos los issues de apache/helix

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.