ResourceConfigCache Staleness causes same workflow to be deleted multiple times.

Aperta
  • Qualcosa non va?
#2,957 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

  • #2958 di @himanshukandwal — chiusa senza merge

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
java
Ambito
distributed-systems

Direzione di ricerca

Traccia TaskDriver.delete attraverso Helix Controller Pipeline, concentrandoti sugli aggiornamenti della cache di ResourceConfig e sull’ordine degli eventi ResourceConfigChange e OnDemandRebalance. Confronta come vengono preparati ResourceConfig e TaskDataCache dopo una cancellazione; il lavoro è completato quando un workflow cancellato non viene reintrodotto e viene cancellato una sola volta, anche durante sequenze di eventi in un cluster occupato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Describe the bug

When a Workflow is marked as deleted (TargetState.DELETE) via TaskDriver.delete API, then the workflow is deleted by the Helix Controller Pipeline in the next run. With deletion, the entries from ZK are deleted and the TaskDataCache is also updated. However, the other entries for this workflow are still present in ResourceConfig (Base) cache.

Usually, the next event which is in the pipeline is the ResourceConfigChange Event (as the resource config was deleted), which takes care of the resourceConfig cache update, but in case of a very busy cluster, other change event can be already earlier in the pipeline than the scheduled ResourceConfigChange event. Now, the ResourceConfig cache is updated selectively and not always, so the ResourceConfig cache keeps the deleted workflow entries, and when the Workflow (TaskDataCache) is prepared, these (previously deleted) workflow entries comes back in again. This causes, same workflows to be deleted multiple times. (Until we see the ResourceConfigChange or OnDemandRebalance event).

TLDR: In Busy Cluster, the resourceConfig cache can take time to be eventually consistent and this causes duplicate deletes of the same workflow.

Impact: Some customers delete and re-create workflow with same name and this behavior causes the recently deleted workflow to be deleted again (unexpectedly).

To Reproduce

Since this happens in a busy cluster where a lot of events are happening (indeterministically) so its not possible to reproduce this behavior. This however is evident in the cluster logs.

Expected behavior

Deletion should happen once only, and ResourceConfig cache should be updated immediately to serve the next events in a reliable and deterministic manner.

Additional context

Add any other context about the problem here.

Lingua principale
Java
Stelle
505
Fork
258
Merge medio
4g 20h
PR unite (30g)
3

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/helix

Tutte le issue di apache/helix

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.