validator: make /health report whether the client can do its duties

Aberta
#833 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 6 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
rust

Direção de pesquisa

Start with crates/validator/src/metrics.rs:664-674 and the state updates in crates/validator/src/lib.rs and crates/validator/src/slashing_protection/mod.rs; read docs/metrics.md and the Metrics section in docs/cli.md for the documentation changes. Decide the open questions about probe semantics, empty keys, degraded state and signer checks before implementation. Done means the endpoint and health gauge agree, the documented checks and status behavior are covered by the listed tests, and beacon_node_available recovers after a successful duty.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

The validator client's /health always answers 200 {"status":"healthy","service":"ethlambda-validator"}. It says the process is up and nothing else. A client whose beacon nodes are all down, or that loaded no keys, or whose slashing-protection writes are failing, answers the same way. Kubernetes probes, load balancers and uptime checks that read it are told everything is fine while no duty is being done.

Code references are at main @ 5d8628e9.

What happens today

  • The handler is a constant. crates/validator/src/metrics.rs:664-669 returns the same body with no input. It is served on --metrics-port next to /metrics (metrics.rs:671-674).
  • It is bound early on purpose. crates/validator/src/lib.rs:217-232 binds it before any beacon node call, so that "an operator can tell 'still starting, node unreachable' apart from 'crashed with no HTTP surface at all'". It does tell those two apart. It cannot tell "still starting" from "running and attesting", because both answer healthy.
  • The code already knows it is not enough. run_until_stopped's doc (lib.rs:575-582) says that carrying on with a poisoned slashing-protection record "would leave a client that answers /health and does not attest, which nothing that supervises it has a reason to touch". That case is handled by exiting the process (lib.rs:594-612). The other ways to stop doing duties are not.
  • The state exists, as metrics.
    • ethlambda_validator_validators_loaded (set_validators_loaded, lib.rs:212).
    • ethlambda_validator_validators_resolved and duties_held, set in refresh_epoch (lib.rs:1026, lib.rs:1050).
    • ethlambda_validator_beacon_node_available (metrics.rs:308-314).
    • ethlambda_validator_slashing_protection_write_failures_total (metrics.rs:367, counted at slashing_protection/mod.rs:871).
  • beacon_node_available cannot simply be mirrored. Its help text says "1 when a beacon node answered the last duty refresh or attestation". In the code:
    • It is set to 1 only at the end of a successful refresh_epoch (lib.rs:1051), once per epoch.
    • It is set to 0 on any failed refresh, subscription, attestation, sync committee message or PTC vote (lib.rs:438, :497, :698, :807, :838).
    • So one failed attestation keeps it at 0 until the next epoch's refresh, up to 6.4 minutes on mainnet, even if every later attestation is accepted. A successful attestation never sets it back.
    • With no keys loaded it never becomes 1, because refresh_epoch returns before the line that sets it (lib.rs:1001-1002).
    • It is one value for all nodes. Nothing records which configured node is up.

What the client cannot report today

These are the states in which the process is up and no duty is done, or duties are done badly:

  1. No beacon node usable. Every configured node is unreachable, syncing or optimistic. refresh_epoch keeps the previous schedule and retries (lib.rs:436-439), and the attestation path fails each slot.
  2. No keys. store.is_empty() only logs "No validators loaded; the client will idle" once at startup (lib.rs:209-210).
  3. Keys loaded but none resolved. validators_resolved is 0 while validators_loaded is not. That is normal before activation, and wrong on a node that resolves nothing because it is on another network (see #831).
  4. Slashing protection refusing every signature.
    • A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (slashing_protection/mod.rs:860-879). The next slot tries again and fails the same way.
    • A record held past RECORD_WAIT (3 s, slashing_protection/mod.rs:118) refuses the duty with "a write may be stuck" (slashing_protection/mod.rs:171-178).
    • Neither stops the process. A poisoned record does, so /health does not need a state for it.
  5. Starting up. Between binding the listener and the first successful refresh, nothing has been checked yet.

What other clients report

Client Endpoint What it reports Read at
Lighthouse GET /lighthouse/health and GET /lighthouse/beacon/health on the validator client's HTTP API, behind its API token (validator_client/http_api/src/lib.rs:306-316, :433-466, :1425-1428) The first: host and process statistics. The second: every beacon node's health tier (sync distance, optimistic, execution client) or the reason it is unusable, the same state its failover ranks by. Both answer with a JSON body for the reader to interpret. unstable @ 11813cfb
Prysm GET /healthz on the monitoring port, which the validator client registers unless --disable-monitoring is set (validator/node/node.go:394-399, monitoring/prometheus/service.go:48, :61-91) 503 when any registered service reports an error. The validator service only reports whether it holds a connection object ("no connection to beacon RPC", validator/client/service.go:288-293). A connected node that is syncing passes. develop @ 0c1e3d24
Teku None. The validator client's REST API serves keymanager routes only (validator/client/src/main/java/tech/pegasys/teku/validator/client/restapi/ValidatorRestApi.java:122-146). master @ 62a54b96
Beacon API, for beacon nodes GET /eth/v1/node/health (ethereum/beacon-APIs @ 6bc8c7bd, apis/node/health.yaml:6-24) 200 ready, 206 syncing or optimistic, 503 not initialized or having issues. "Useful for load balancers."

No validator client read here reports keys or slashing-protection state through a health endpoint. Lighthouse's per-node view is the closest to the beacon_nodes check proposed below, and the Beacon API's status codes are the convention a probe already understands.

Proposal

Keep /health cheap and local: it reads state the duty loop already maintains and makes no beacon node request of its own. Answer 200 when the client can do its duties and 503 when it cannot, with a JSON body that says why.

{
  "status": "unhealthy",
  "service": "ethlambda-validator",
  "checks": {
    "beacon_nodes": { "usable": 0, "configured": 2 },
    "validators": { "loaded": 64, "resolved": 64 },
    "slashing_protection": { "last_write_ok": false },
    "started": true
  }
}
  • beacon_nodes. At least one configured node answered its last probe, is not syncing and not optimistic. This needs per-node state, which #830 adds. Until that lands, a smaller version can use the outcome of the last beacon node call of any kind, with a success setting it back.
  • validators. loaded and resolved, reported. resolved = 0 does not fail the check, since keys waiting for activation are normal. Whether loaded = 0 does is open (see "To decide").
  • slashing_protection. The last commit succeeded and the last access did not time out on RECORD_WAIT. A failed write fails the check until a later write succeeds.
  • started. false until the first successful refresh_epoch, so a probe can tell "starting" from "running".

Alongside it:

  • Fix beacon_node_available. Set it from the same per-node state, or at least to 1 on every successful duty call, so it matches its help text. Document the change in docs/metrics.md.
  • Add a ethlambda_validator_healthy gauge with the same value as the endpoint, so a Prometheus alert and a probe cannot disagree.
  • Document the endpoint in docs/cli.md's Metrics section, which does not mention /health today.

To decide

  • One endpoint or two. A liveness probe that fails while the beacon node is down makes Kubernetes restart the validator client, which does not help and adds a restart to the outage. Kubernetes separates liveness from readiness for this reason. Options: keep /health as liveness (always 200 while the process serves) and add /ready with the checks above, or make /health the checked one and tell operators not to use it as a liveness probe.
  • A degraded state. Some nodes down but one usable, or some keys unresolved. 200 with the detail in the body, or a separate status like the beacon API's 206.
  • Whether remote signers count. A Web3Signer that does not answer loses its keys' signatures (SIGNING_BUDGET, crates/validator/src/signing.rs:120). An upcheck per configured signer would put that in the body. It is a network call, so it would need its own cadence.
  • No keys. A client started empty to wait for a keymanager import is idle on purpose. Whether loaded = 0 answers 503, or 200 with the count in the body.
  • Doppelganger. #807 adds keys that are loaded but withheld while the check runs. Whether those count as loaded for the check.

Tests

  • The handler returns 503 before the first refresh and 200 after one, driven through MockBeaconNode.
  • Every configured node failing turns it to 503; one recovering turns it back.
  • A slashing-protection write that fails (an unwritable directory) turns it to 503; a later successful write clears it.
  • An empty key store reports validators.loaded = 0.
  • beacon_node_available returns to 1 on the first successful duty after a failure.
Linguagem predominante
Rust
Estrelas
86
Forks
34
Merge médio
9d 20h
PRs com merge (30d)
77

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de lambdaclass/ethlambda

Todas as issues de lambdaclass/ethlambda

Issues semelhantes

Mais issues de Rust

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.