grafana/mimir

Add alert for ring members in non-active state

Open

#2342 aperta il 8 lug 2022

Vedi su GitHub
 (0 commenti) (0 reazioni) (1 assegnatario)Go (808 fork)auto 404
enhancementgood first issuemonitoring/alerts

Metriche repository

Star
 (5181 star)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

As an extension to #1960, we should consider adding an alert when ring members are in a non-active state for an unexpected amount of time.

I had suggested something like min by(cluster, namespace) (cortex_ring_members{name="distributor",state!="ACTIVE"}) > 0 for longer than 30min, but feedback from @pracucci highlighted:

  • For some components not supporting the auto-forget feature (e.g. distributors till very recent times) it's expected to have non-ACTIVE ring members for a long time and everything works as expected anyway
  • I think the query you're suggesting would suffer the false positives issue during rollouts

Guida contributor