notifier: group state is mutated under read locks from concurrent goroutines (potential fatal concurrent map write)

Aperta
#869 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
go
Ambito
backend

Direzione di ricerca

Inizia in coordinator.go, in corrispondenza di responseLoop (intorno alla riga 407), checkAndSendResponseToModules e notifyModule, quindi confronta la correzione relativa al locking in issue #867. Determina come rendere sicuro il check-and-set di ID, Start e LastNotify senza lasciare l’invio delle notifiche sotto un lock inutilmente ampio; il lavoro è completato quando le risposte concorrenti non possono entrare in race né provocare scritture fatali nelle map.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

checkAndSendResponseToModules is spawned as a goroutine per evaluator response (responseLoop, coordinator.go:407), so multiple instances run concurrently by design. Each holds only read locks (nc.clusterLock.RLock() + cluster.Lock.RLock()) while mutating shared per-group state:

  • cgroup.ID and cgroup.Start are written at coordinator.go:430-434 and 454-458 under the read lock.
  • notifyModule (called synchronously from there, still under the same read locks) writes to the cgroup.LastNotify map at coordinator.go:548 and 566.

Read locks don't exclude each other, so two concurrent responses for the same group can write cgroup.ID/Start racily and — worse — perform concurrent writes to the same LastNotify map. Concurrent map writes are a fatal, unrecoverable runtime error in Go (fatal error: concurrent map writes), which would take down the whole Burrow process.

Two responses for the same group can be in flight at once, e.g. when two sendEvaluatorRequests loops briefly overlap after a ZooKeeper session bounce (manageEvalLoop starts a new loop once it reacquires the ZK lock while the old one is still mid-iteration), or whenever response processing is slower than the evaluation interval.

Why not just a PR

The minimal fix — upgrading cluster.Lock.RLock() to Lock() in checkAndSendResponseToModules — is a one-line change, but it would hold the cluster's write lock across module.Notify() calls (HTTP/email sends), serializing notification dispatch per cluster and blocking the evaluation loop for that cluster while a slow notifier runs. Alternatives:

  1. Accept the serialization (correctness over throughput; notify calls are usually fast).
  2. Restructure so the LastNotify/ID/Start check-and-set happens under a short write lock and Notify() is called after release (the status/startTime/eventID are already passed by value).
  3. Give consumerGroup its own small mutex for ID/Start/LastNotify.

Option 2 seems cleanest but touches the send-once/send-interval logic, so I wanted maintainer input on the preferred direction before sending a PR. Happy to implement whichever is preferred.

Related: #867 fixes the same pattern (write under read lock) in sendEvaluatorRequests, where the fix has no locking tradeoff.

Burrow version: master (60d5782)

Lingua principale
Go
Stelle
4k
Fork
816
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di linkedin/Burrow

Tutte le issue di linkedin/Burrow

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.