Controller endlessly creates BEST_POSSIBLE znodes during WAGED migration

Aperta
  • Qualcosa non va?
#2,949 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
java
Ambito
distributed-systems

Direzione di ricerca

Inizia con i passaggi per riprodurre il problema e il codice di test in https://github.com/GrantPSpencer/helix/pull/45. Traccia emergencyRebalance, _assignmentManager.getBestPossibleAssignment, WagedRebalanceUtil.calculateAssignment e _assignmentMetadataStore.asyncUpdateBestPossibleAssignmentCache attraverso il flusso di migrazione WAGED. Il lavoro è completato quando la migrazione non valida smette di riattivare il ribilanciamento e non crea BEST_POSSIBLE znodes all’infinito; il ribilanciamento dovrebbe fallire come descritto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Describe the bug

Under specific circumstances, WAGED migration will cause the controller to endleslly write a new znode under BEST_POSSIBLE.

To Reproduce

Some test code to reproduce the issue:
https://github.com/GrantPSpencer/helix/pull/45

  1. Have CRUSHED resources
  2. Disable a node's partition on all instances in the cluster (essentially making no valid assignment)
  3. Switched to WAGED
  4. Observe Behavior
Expected behavior

Rebalance should fail and new best possible nodes should not be endlessly created

The fix is not straight forward. See below for my thoughts on what triggers this. I think the true solution for this might be to have partialRebalance correctly fail to generate a mapping, rather than tweaking our persist assignment logic.

Additional context

Say we have resource_0 and resource_1, both currently using CRUSHED. We have disabled resource_0's only partition on all instances in the cluster. The current state will be offline for all nodes for resource_0 while using CRUSHED. Then we switched the resource to WAGED:

WAGED rebalancer's emergencyRebalance _assignmentManager.getBestPossibleAssignment(, which looks at the in-memory (or zk-stored) assignment and fills in any missing resources with the current states (resource_0 still has all OFFLINE for current states)
A best possible with resource_0 and resource_1's current states are stored in ZK and memory.

We then trigger a partial rebalance. When partialRebalance calls WagedRebalanceUtil.calculateAssignment it is passing in a clusterModel that has removed resource_0 from the replica map so toBeAssignedReplicas does not include resource_0. It then produces a result it sees as valid that only includes resource_1. We then call _assignmentMetadataStore.asyncUpdateBestPossibleAssignmentCache which stores the result in memory and triggers an onDemand rebalance.

We start the onDemand rebalance and our in-memory best possible only has an assignment for resource_1. In emergency rebalance, emergencyRebalance once again calls _assignmentManager.getBestPossibleAssignment( and it takes the best possible that only has resource_1 and fills in the blanks for resource_0 by combining it with the current state, so it creates a new map with assignments for resource_0 and resource_1 where all resource_0 assignments are offline.
We then call persistBestPossibleAssignment, which writes the mapping to zookeeper and stores it in memory.

We then do another partial rebalance, which computes a mapping with only resource_1, persists that into memory, and triggers an onDemand rebalance. I believe this will occur endlessly until a node goes down, which leads to calling calculateAssignment( instead of getBestPossibleAssignment( and persist an assignment that only includes resource_1 to Z (rather than just in memory previously)

Lingua principale
Java
Stelle
505
Fork
258
Merge medio
4g 20h
PR unite (30g)
3

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/helix

Tutte le issue di apache/helix

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.