CSM Alerts & Monitoring: add a section on node/client health alerting (catch a hung client before missed duties)

Open
  • Is something wrong?
#1,005 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
grafana, prometheus
Domain
documentation, observability

Research direction

Start with the CSM Alerts & Monitoring page linked in the issue, especially its Advanced section and existing Prometheus/Grafana setup. Document client-agnostic node-health signals, Alertmanager rules, and the need for off-machine paging; include hosted options only under neutral criteria if appropriate. Done means the section covers the proposed signals and explains how operators can act before missed duties.

Written by the indexing model from the issue text.

Description

Gap

The CSM Alerts & Monitoring section covers dashboards (Beginner/Intermediate), a local Grafana stack (Advanced) and on-chain event alerts (Expert). Nothing yet covers alerts on the node's own health: execution or consensus client down, hung, or stuck syncing, falling peers, a disk about to fill.

For CSM operators those are the most common causes of missed duties, and every tool currently listed only sees them afterwards, as a missed attestation or a penalty. The Advanced level sets up Grafana, but dashboards don't page anyone, and a stack on the same machine goes down with it.

Proposal

A short subsection (under Advanced, or a new "Node health alerts" page) covering:

  1. What to alert on, client-agnostic: EL/CL API unreachable, head not advancing, sync distance growing, peers low, disk free below a threshold, clock drift.
  2. Self-hosted option: Alertmanager on top of the Prometheus that the stacks in the Advanced section already run, with example rules for the signals above, and a note to run it (or a dead-man switch) off the validator machine so it can page when the box is down.
  3. Hosted options: external services that page you even when the machine is gone.

Disclosure and offer

I maintain NodeBeat, an open-source (Apache-2.0), read-only agent with hosted alerting that fits under (3). I understand if you'd rather not list third-party services. The section is useful without it, and I'm happy to write the PR with only (1) and (2). If you're open to listing hosted options, I'd suggest neutral criteria (read-only, open-source agent, no key access) and listing every service that meets them, not just mine.

Dominant language
JavaScript
Stars
61
Forks
82
Avg merge
1d 18h
Merged PRs (30d)
16

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lidofinance/docs

All issues in lidofinance/docs

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.