Helix Controller Does Not Rebalance Partitions After A Participant Loss

Open
  • Is something wrong?
#3,049 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, docker, java
Domain
cloud, distributed-systems, infrastructure

Research direction

Start by tracing Apache Helix's FULL_AUTO rebalance handling for the OnlineOffline state model during a forced AWS ECS Spot termination. Compare LIVEINSTANCES and EXTERNALVIEW after the node disappears, then compare the controller behavior with the manual service restart that restores assignments. Done means partitions formerly owned by the stopped node are automatically reassigned to live nodes without restarting all services.

Written by the indexing model from the issue text.

Description

bug
Problem

We are using Apache Helix to distribute a resource across multiple services - all registered under the same Helix cluster name and The system is deployed on AWS ECS Fargate using Docker containers.

Each service uses the OnlineOffline state model, with 1 replica per partition and the FULL_AUTO rebalance mode.

When all services start, Helix assigns partitions as expected and all services consume their assigned partitions. The problem occurs when a service is restarted - due to an AWS Spot Termination which appear to forcefully kill the container, bypassing the logic we have to handle SIGTERM

So when a service is restarted from the cluster, the partition previously assigned to the now-stopped node are not reassigned to new live nodes. As a result, that partition remains Idle, even though other nodes are available and registered in live instances.

I have tried with both Apache Helix version 0.9.9 and the latest 1.4.3, but the problem still persists in both versions. Also When a node is removed from the cluster, there are no logs from the controller instance indicating that it is attempting to rebalance the partitions.

Additional Context After ZooKeeper Inspection
  • /LIVEINSTANCES is perfectly listing only actively running nodes.
  • But in /EXTERNALVIEW for that partition still shows the stopped node as ONLINE and does not reassign it to a live node.
  • This mismatch persists even after new nodes are registered as live instances.
Expected behavior

Helix should detect that the node is no longer live and automatically reassign its partitions to other available nodes in the cluster.

What worked for us

But I have been able to temporarily resolve the issue by performing a manual restart of all services register under that HELIX CLUSTER. After the manual restart, all partitions are perfectly reassigned.

I will be active here and will respond to all your replies as soon as possible and I am happy to provide any additional information or logs that would help with debugging this issue.

Dominant language
Java
Stars
505
Forks
258
Avg merge
4d 20h
Merged PRs (30d)
3

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/helix

All issues in apache/helix

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.