validator: make /health report whether the client can do its duties
維護者通常 6 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 35/100
- Issue 類型
- 功能
- 描述清晰度
- 基本清楚
- 活躍度
- 活躍
- 技術堆疊
- rust
- 領域
- backend, observability
研究方向
Start with crates/validator/src/metrics.rs:664-674 and the state updates in crates/validator/src/lib.rs and crates/validator/src/slashing_protection/mod.rs; read docs/metrics.md and the Metrics section in docs/cli.md for the documentation changes. Decide the open questions about probe semantics, empty keys, degraded state and signer checks before implementation. Done means the endpoint and health gauge agree, the documented checks and status behavior are covered by the listed tests, and beacon_node_available recovers after a successful duty.
由索引模型根據 Issue 內容生成。
描述
The validator client's /health always answers 200 {"status":"healthy","service":"ethlambda-validator"}. It says the process is up and nothing else. A client whose beacon nodes are all down, or that loaded no keys, or whose slashing-protection writes are failing, answers the same way. Kubernetes probes, load balancers and uptime checks that read it are told everything is fine while no duty is being done.
Code references are at main @ 5d8628e9.
What happens today
- The handler is a constant.
crates/validator/src/metrics.rs:664-669returns the same body with no input. It is served on--metrics-portnext to/metrics(metrics.rs:671-674). - It is bound early on purpose.
crates/validator/src/lib.rs:217-232binds it before any beacon node call, so that "an operator can tell 'still starting, node unreachable' apart from 'crashed with no HTTP surface at all'". It does tell those two apart. It cannot tell "still starting" from "running and attesting", because both answerhealthy. - The code already knows it is not enough.
run_until_stopped's doc (lib.rs:575-582) says that carrying on with a poisoned slashing-protection record "would leave a client that answers/healthand does not attest, which nothing that supervises it has a reason to touch". That case is handled by exiting the process (lib.rs:594-612). The other ways to stop doing duties are not. - The state exists, as metrics.
ethlambda_validator_validators_loaded(set_validators_loaded,lib.rs:212).ethlambda_validator_validators_resolvedandduties_held, set inrefresh_epoch(lib.rs:1026,lib.rs:1050).ethlambda_validator_beacon_node_available(metrics.rs:308-314).ethlambda_validator_slashing_protection_write_failures_total(metrics.rs:367, counted atslashing_protection/mod.rs:871).
beacon_node_availablecannot simply be mirrored. Its help text says "1 when a beacon node answered the last duty refresh or attestation". In the code:- It is set to
1only at the end of a successfulrefresh_epoch(lib.rs:1051), once per epoch. - It is set to
0on any failed refresh, subscription, attestation, sync committee message or PTC vote (lib.rs:438,:497,:698,:807,:838). - So one failed attestation keeps it at
0until the next epoch's refresh, up to 6.4 minutes on mainnet, even if every later attestation is accepted. A successful attestation never sets it back. - With no keys loaded it never becomes
1, becauserefresh_epochreturns before the line that sets it (lib.rs:1001-1002). - It is one value for all nodes. Nothing records which configured node is up.
- It is set to
What the client cannot report today
These are the states in which the process is up and no duty is done, or duties are done badly:
- No beacon node usable. Every configured node is unreachable, syncing or optimistic.
refresh_epochkeeps the previous schedule and retries (lib.rs:436-439), and the attestation path fails each slot. - No keys.
store.is_empty()only logs "No validators loaded; the client will idle" once at startup (lib.rs:209-210). - Keys loaded but none resolved.
validators_resolvedis0whilevalidators_loadedis not. That is normal before activation, and wrong on a node that resolves nothing because it is on another network (see #831). - Slashing protection refusing every signature.
- A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (
slashing_protection/mod.rs:860-879). The next slot tries again and fails the same way. - A record held past
RECORD_WAIT(3 s,slashing_protection/mod.rs:118) refuses the duty with "a write may be stuck" (slashing_protection/mod.rs:171-178). - Neither stops the process. A poisoned record does, so
/healthdoes not need a state for it.
- A write that fails (a full disk, a read-only remount) refuses that duty's signature and counts a failure (
- Starting up. Between binding the listener and the first successful refresh, nothing has been checked yet.
What other clients report
| Client | Endpoint | What it reports | Read at |
|---|---|---|---|
| Lighthouse | GET /lighthouse/health and GET /lighthouse/beacon/health on the validator client's HTTP API, behind its API token (validator_client/http_api/src/lib.rs:306-316, :433-466, :1425-1428) |
The first: host and process statistics. The second: every beacon node's health tier (sync distance, optimistic, execution client) or the reason it is unusable, the same state its failover ranks by. Both answer with a JSON body for the reader to interpret. | unstable @ 11813cfb |
| Prysm | GET /healthz on the monitoring port, which the validator client registers unless --disable-monitoring is set (validator/node/node.go:394-399, monitoring/prometheus/service.go:48, :61-91) |
503 when any registered service reports an error. The validator service only reports whether it holds a connection object ("no connection to beacon RPC", validator/client/service.go:288-293). A connected node that is syncing passes. |
develop @ 0c1e3d24 |
| Teku | None. The validator client's REST API serves keymanager routes only (validator/client/src/main/java/tech/pegasys/teku/validator/client/restapi/ValidatorRestApi.java:122-146). |
master @ 62a54b96 |
|
| Beacon API, for beacon nodes | GET /eth/v1/node/health (ethereum/beacon-APIs @ 6bc8c7bd, apis/node/health.yaml:6-24) |
200 ready, 206 syncing or optimistic, 503 not initialized or having issues. "Useful for load balancers." |
No validator client read here reports keys or slashing-protection state through a health endpoint. Lighthouse's per-node view is the closest to the beacon_nodes check proposed below, and the Beacon API's status codes are the convention a probe already understands.
Proposal
Keep /health cheap and local: it reads state the duty loop already maintains and makes no beacon node request of its own. Answer 200 when the client can do its duties and 503 when it cannot, with a JSON body that says why.
{
"status": "unhealthy",
"service": "ethlambda-validator",
"checks": {
"beacon_nodes": { "usable": 0, "configured": 2 },
"validators": { "loaded": 64, "resolved": 64 },
"slashing_protection": { "last_write_ok": false },
"started": true
}
}
beacon_nodes. At least one configured node answered its last probe, is not syncing and not optimistic. This needs per-node state, which #830 adds. Until that lands, a smaller version can use the outcome of the last beacon node call of any kind, with a success setting it back.validators.loadedandresolved, reported.resolved = 0does not fail the check, since keys waiting for activation are normal. Whetherloaded = 0does is open (see "To decide").slashing_protection. The last commit succeeded and the last access did not time out onRECORD_WAIT. A failed write fails the check until a later write succeeds.started.falseuntil the first successfulrefresh_epoch, so a probe can tell "starting" from "running".
Alongside it:
- Fix
beacon_node_available. Set it from the same per-node state, or at least to1on every successful duty call, so it matches its help text. Document the change indocs/metrics.md. - Add a
ethlambda_validator_healthygauge with the same value as the endpoint, so a Prometheus alert and a probe cannot disagree. - Document the endpoint in
docs/cli.md's Metrics section, which does not mention/healthtoday.
To decide
- One endpoint or two. A liveness probe that fails while the beacon node is down makes Kubernetes restart the validator client, which does not help and adds a restart to the outage. Kubernetes separates liveness from readiness for this reason. Options: keep
/healthas liveness (always200while the process serves) and add/readywith the checks above, or make/healththe checked one and tell operators not to use it as a liveness probe. - A degraded state. Some nodes down but one usable, or some keys unresolved.
200with the detail in the body, or a separate status like the beacon API's206. - Whether remote signers count. A Web3Signer that does not answer loses its keys' signatures (
SIGNING_BUDGET,crates/validator/src/signing.rs:120). Anupcheckper configured signer would put that in the body. It is a network call, so it would need its own cadence. - No keys. A client started empty to wait for a keymanager import is idle on purpose. Whether
loaded = 0answers503, or200with the count in the body. - Doppelganger. #807 adds keys that are loaded but withheld while the check runs. Whether those count as loaded for the check.
Tests
- The handler returns
503before the first refresh and200after one, driven throughMockBeaconNode. - Every configured node failing turns it to
503; one recovering turns it back. - A slashing-protection write that fails (an unwritable directory) turns it to
503; a later successful write clears it. - An empty key store reports
validators.loaded = 0. beacon_node_availablereturns to1on the first successful duty after a failure.
- 主要語言
- Rust
- 星號
- 86
- 分支
- 34
- 平均合併
- 9 天 20 小時
- 30 天內合併 PR
- 77
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
lambdaclass/ethlambda 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 76/100
lambdaclass/ethlambda#850 ·
維護者通常 6 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
lambdaclass/ethlambda#777 · 1 個 reaction ·
維護者通常 6 天內回覆
-
Run all gossip validation tests可能重新可做 @dicethedev 於 9 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 62/100
lambdaclass/ethlambda#725 · 2 則留言 ·
維護者通常 6 天內回覆
-
Handle malformed DB data可能重新可做 @dicethedev 於 115 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 76/100
lambdaclass/ethlambda#529 · 4 則留言 ·
維護者通常 6 天內回覆
-
Document architectural choices可能重新可做 @dicethedev 於 115 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 68/100
lambdaclass/ethlambda#521 · 2 則留言 ·
維護者通常 6 天內回覆
查看 lambdaclass/ethlambda 的全部 Issue
相似的 Issue
-
難度 2/5 1-3 小時 新手友好度 84/100
antithesishq/bombadil#473 ·
維護者通常 2 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 72/100
libracoder/kiri#669 ·
維護者通常 1 天內回覆
-
難度 1/5 1 小時以內 新手友好度 88/100
維護者通常 5 天內回覆
-
autofixer: follow-up
難度 2/5 1-3 小時 新手友好度 75/100
tursodatabase/turso#12217 ·
維護者通常 1 天內回覆
-
Declare the CMS schema library in the optional recipients extra可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉area:python in-progress
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 1 天內回覆