Controller endlessly creates BEST_POSSIBLE znodes during WAGED migration

未关闭
  • 哪里不对？
#2,949 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
停滞
技术栈
java
领域
distributed-systems

调研方向

从 https://github.com/GrantPSpencer/helix/pull/45 中的复现步骤和测试代码开始。沿着 WAGED 迁移流程跟踪 emergencyRebalance、_assignmentManager.getBestPossibleAssignment、WagedRebalanceUtil.calculateAssignment 和 _assignmentMetadataStore.asyncUpdateBestPossibleAssignmentCache。完成的标准是：无效迁移不再重新触发 rebalance，也不会无休止地创建 BEST_POSSIBLE znodes；rebalance 应按描述失败。

由索引模型根据 Issue 内容生成。

描述

bug
Describe the bug

Under specific circumstances, WAGED migration will cause the controller to endleslly write a new znode under BEST_POSSIBLE.

To Reproduce

Some test code to reproduce the issue:
https://github.com/GrantPSpencer/helix/pull/45

  1. Have CRUSHED resources
  2. Disable a node's partition on all instances in the cluster (essentially making no valid assignment)
  3. Switched to WAGED
  4. Observe Behavior
Expected behavior

Rebalance should fail and new best possible nodes should not be endlessly created

The fix is not straight forward. See below for my thoughts on what triggers this. I think the true solution for this might be to have partialRebalance correctly fail to generate a mapping, rather than tweaking our persist assignment logic.

Additional context

Say we have resource_0 and resource_1, both currently using CRUSHED. We have disabled resource_0's only partition on all instances in the cluster. The current state will be offline for all nodes for resource_0 while using CRUSHED. Then we switched the resource to WAGED:

WAGED rebalancer's emergencyRebalance _assignmentManager.getBestPossibleAssignment(, which looks at the in-memory (or zk-stored) assignment and fills in any missing resources with the current states (resource_0 still has all OFFLINE for current states)
A best possible with resource_0 and resource_1's current states are stored in ZK and memory.

We then trigger a partial rebalance. When partialRebalance calls WagedRebalanceUtil.calculateAssignment it is passing in a clusterModel that has removed resource_0 from the replica map so toBeAssignedReplicas does not include resource_0. It then produces a result it sees as valid that only includes resource_1. We then call _assignmentMetadataStore.asyncUpdateBestPossibleAssignmentCache which stores the result in memory and triggers an onDemand rebalance.

We start the onDemand rebalance and our in-memory best possible only has an assignment for resource_1. In emergency rebalance, emergencyRebalance once again calls _assignmentManager.getBestPossibleAssignment( and it takes the best possible that only has resource_1 and fills in the blanks for resource_0 by combining it with the current state, so it creates a new map with assignments for resource_0 and resource_1 where all resource_0 assignments are offline.
We then call persistBestPossibleAssignment, which writes the mapping to zookeeper and stores it in memory.

We then do another partial rebalance, which computes a mapping with only resource_1, persists that into memory, and triggers an onDemand rebalance. I believe this will occur endlessly until a node goes down, which leads to calling calculateAssignment( instead of getBestPossibleAssignment( and persist an assignment that only includes resource_1 to Z (rather than just in memory previously)

主要语言
Java
星标
505
派生
258
平均合并
4 天 20 小时
30 天内合并 PR
3

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/helix 的其他 Issue

查看 apache/helix 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。