grafana/mimir

Add alert for ring members in non-active state

Open

#2,342 建立於 2022年7月8日

在 GitHub 查看
 (0 留言) (0 反應) (1 負責人)Go (808 fork)auto 404
enhancementgood first issuemonitoring/alerts

倉庫指標

Star
 (5,181 star)
PR 合併指標
 (PR 指標待抓取)

描述

As an extension to #1960, we should consider adding an alert when ring members are in a non-active state for an unexpected amount of time.

I had suggested something like min by(cluster, namespace) (cortex_ring_members{name="distributor",state!="ACTIVE"}) > 0 for longer than 30min, but feedback from @pracucci highlighted:

  • For some components not supporting the auto-forget feature (e.g. distributors till very recent times) it's expected to have non-ACTIVE ring members for a long time and everything works as expected anyway
  • I think the query you're suggesting would suffer the false positives issue during rollouts

貢獻者指南