validator: make /health report whether the client can do its duties
メンテナーはふだん 6 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
- issue の種類
- 機能追加
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- rust
- 領域
- backend, observability
調査の方向性
Start with crates/validator/src/metrics.rs:664-674 and the state updates in crates/validator/src/lib.rs and crates/validator/src/slashing_protection/mod.rs; read docs/metrics.md and the Metrics section in docs/cli.md for the documentation changes. Decide the open questions about probe semantics, empty keys, degraded state and signer checks before implementation. Done means the endpoint and health gauge agree, the documented checks and status behavior are covered by the listed tests, and beacon_node_available recovers after a successful duty.
索引モデルが issue の本文から書いたものです。
説明
The validator client's /health always answers 200 {"status":"healthy","service":"ethlambda-validator"}. It says the process is up and nothing else. A client whose beacon nodes are all down, or that loaded no keys, or whose slashing-protection writes are failing, answers the same way. Kubernetes probes, load balancers and uptime checks that read it are told everything is fine while no duty is being done.
Code references are at main @ 5d8628e9.
What happens today
- The handler is a constant.
crates/validator/src/metrics.rs:664-669returns the same body with no input. It is served on--metrics-portnext to/metrics(metrics.rs:671-674). - It is bound early on purpose.
crates/validator/src/lib.rs:217-232binds it before any beacon node call, so that "an operator can tell 'still starting, node unreachable' apart from 'crashed with no HTTP surface at all'". It does tell those two apart. It cannot tell "still starting" from "running and attesting", because both answerhealthy. - The code already knows it is not enough.
run_until_stopped's doc (lib.rs:575-582) says that carrying on with a poisoned slashing-protection record "would leave a client that answers/healthand does not attest, which nothing that supervises it has a reason to touch". That case is handled by exiting the process (lib.rs:594-612). The other ways to stop doing duties are not. - The state exists, as metrics.
ethlambda_validator_validators_loaded(set_validators_loaded,lib.rs:212).ethlambda_validator_validators_resolvedandduties_held, set inrefresh_epoch(lib.rs:1026,lib.rs:1050).ethlambda_validator_beacon_node_available(metrics.rs:308-314).ethlambda_validator_slashing_protection_write_failures_total(metrics.rs:367, counted atslashing_protection/mod.rs:871).
beacon_node_availablecannot simply be mirrored. Its help text says "1 when a beacon node answered the last duty refresh or attestation". In the code:- It is set to
1only at the end of a successfulrefresh_epoch(lib.rs:1051), once per epoch. - It is set to
0on any failed refresh, subscription, attestation, sync committee message or PTC vote (lib.rs:438,:497,:698,:807,:838). - So one failed attestation keeps it at
0until the next epoch's refresh, up to 6.4 minutes on mainnet, even if every later attestation is accepted. A successful attestation never sets it back. - With no keys loaded it never becomes
1, becauserefresh_epochreturns before the line that sets it (lib.rs:1001-1002). - It is one value for all nodes. Nothing records which configured node is up.
- It is set to
What the client cannot report today
These are the states in which the process is up and no duty is done, or duties are done badly:
- No beacon node usable. Every configured node is unreachable, syncing or optimistic.
refresh_epochkeeps the previous schedule and retries (lib.rs:436-439), and the attestation path fails each slot. - No keys.
store.is_empty()only logs "No validators loaded; the client will idle" once at startup (lib.rs:209-210). - Keys loaded but none resolved.
validators_resolvedis0whilevalidators_loadedis not. That is normal before activation, and wrong on a node that resolves nothing because it is on another network (see #831). - Slashing protection refusing every signature.
- A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (
slashing_protection/mod.rs:860-879). The next slot tries again and fails the same way. - A record held past
RECORD_WAIT(3 s,slashing_protection/mod.rs:118) refuses the duty with "a write may be stuck" (slashing_protection/mod.rs:171-178). - Neither stops the process. A poisoned record does, so
/healthdoes not need a state for it.
- A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (
- Starting up. Between binding the listener and the first successful refresh, nothing has been checked yet.
What other clients report
| Client | Endpoint | What it reports | Read at |
|---|---|---|---|
| Lighthouse | GET /lighthouse/health and GET /lighthouse/beacon/health on the validator client's HTTP API, behind its API token (validator_client/http_api/src/lib.rs:306-316, :433-466, :1425-1428) |
The first: host and process statistics. The second: every beacon node's health tier (sync distance, optimistic, execution client) or the reason it is unusable, the same state its failover ranks by. Both answer with a JSON body for the reader to interpret. | unstable @ 11813cfb |
| Prysm | GET /healthz on the monitoring port, which the validator client registers unless --disable-monitoring is set (validator/node/node.go:394-399, monitoring/prometheus/service.go:48, :61-91) |
503 when any registered service reports an error. The validator service only reports whether it holds a connection object ("no connection to beacon RPC", validator/client/service.go:288-293). A connected node that is syncing passes. |
develop @ 0c1e3d24 |
| Teku | None. The validator client's REST API serves keymanager routes only (validator/client/src/main/java/tech/pegasys/teku/validator/client/restapi/ValidatorRestApi.java:122-146). |
master @ 62a54b96 |
|
| Beacon API, for beacon nodes | GET /eth/v1/node/health (ethereum/beacon-APIs @ 6bc8c7bd, apis/node/health.yaml:6-24) |
200 ready, 206 syncing or optimistic, 503 not initialized or having issues. "Useful for load balancers." |
No validator client read here reports keys or slashing-protection state through a health endpoint. Lighthouse's per-node view is the closest to the beacon_nodes check proposed below, and the Beacon API's status codes are the convention a probe already understands.
Proposal
Keep /health cheap and local: it reads state the duty loop already maintains and makes no beacon node request of its own. Answer 200 when the client can do its duties and 503 when it cannot, with a JSON body that says why.
{
"status": "unhealthy",
"service": "ethlambda-validator",
"checks": {
"beacon_nodes": { "usable": 0, "configured": 2 },
"validators": { "loaded": 64, "resolved": 64 },
"slashing_protection": { "last_write_ok": false },
"started": true
}
}
beacon_nodes. At least one configured node answered its last probe, is not syncing and not optimistic. This needs per-node state, which #830 adds. Until that lands, a smaller version can use the outcome of the last beacon node call of any kind, with a success setting it back.validators.loadedandresolved, reported.resolved = 0does not fail the check, since keys waiting for activation are normal. Whetherloaded = 0does is open (see "To decide").slashing_protection. The last commit succeeded and the last access did not time out onRECORD_WAIT. A failed write fails the check until a later write succeeds.started.falseuntil the first successfulrefresh_epoch, so a probe can tell "starting" from "running".
Alongside it:
- Fix
beacon_node_available. Set it from the same per-node state, or at least to1on every successful duty call, so it matches its help text. Document the change indocs/metrics.md. - Add a
ethlambda_validator_healthygauge with the same value as the endpoint, so a Prometheus alert and a probe cannot disagree. - Document the endpoint in
docs/cli.md's Metrics section, which does not mention/healthtoday.
To decide
- One endpoint or two. A liveness probe that fails while the beacon node is down makes Kubernetes restart the validator client, which does not help and adds a restart to the outage. Kubernetes separates liveness from readiness for this reason. Options: keep
/healthas liveness (always200while the process serves) and add/readywith the checks above, or make/healththe checked one and tell operators not to use it as a liveness probe. - A degraded state. Some nodes down but one usable, or some keys unresolved.
200with the detail in the body, or a separate status like the beacon API's206. - Whether remote signers count. A Web3Signer that does not answer loses its keys' signatures (
SIGNING_BUDGET,crates/validator/src/signing.rs:120). Anupcheckper configured signer would put that in the body. It is a network call, so it would need its own cadence. - No keys. A client started empty to wait for a keymanager import is idle on purpose. Whether
loaded = 0answers503, or200with the count in the body. - Doppelganger. #807 adds keys that are loaded but withheld while the check runs. Whether those count as loaded for the check.
Tests
- The handler returns
503before the first refresh and200after one, driven throughMockBeaconNode. - Every configured node failing turns it to
503; one recovering turns it back. - A slashing-protection write that fails (an unwritable directory) turns it to
503; a later successful write clears it. - An empty key store reports
validators.loaded = 0. beacon_node_availablereturns to1on the first successful duty after a failure.
- 主要言語
- Rust
- スター
- 86
- フォーク
- 34
- 平均マージ
- 9日 20時間
- マージ済み PR(30日)
- 77
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
lambdaclass/ethlambda のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
lambdaclass/ethlambda#850 ·
メンテナーはふだん 6 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
lambdaclass/ethlambda#777 · リアクション 1 件 ·
メンテナーはふだん 6 日以内に返信
-
Run all gossip validation tests再び着手できるかも @dicethedev が 9 日前に担当しましたが、オープン中のプルリクエストはありません。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
lambdaclass/ethlambda#725 · コメント 2 件 ·
メンテナーはふだん 6 日以内に返信
-
Handle malformed DB data再び着手できるかも @dicethedev が 115 日前に担当しましたが、オープン中のプルリクエストはありません。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
lambdaclass/ethlambda#529 · コメント 4 件 ·
メンテナーはふだん 6 日以内に返信
-
Document architectural choices再び着手できるかも @dicethedev が 115 日前に担当しましたが、オープン中のプルリクエストはありません。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
lambdaclass/ethlambda#521 · コメント 2 件 ·
メンテナーはふだん 6 日以内に返信
lambdaclass/ethlambda の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 84/100
antithesishq/bombadil#473 ·
メンテナーはふだん 2 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
libracoder/kiri#669 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
メンテナーはふだん 5 日以内に返信
-
autofixer: follow-up
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
tursodatabase/turso#12217 ·
メンテナーはふだん 1 日以内に返信
-
Declare the CMS schema library in the optional recipients extra対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンarea:python in-progress
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
yhay81/pyropdf#1005 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信