validator: shut down gracefully on SIGINT and SIGTERM
維護者通常 6 天內回覆
還沒有人認領這個 Issue。
評估
研究方向
Start with bin/ethlambda/src/validator.rs:245-276 and bin/ethlambda/src/shutdown.rs:31, then trace crates/validator/src/lib.rs:561-627 and its run_until_stopped tests. The issue proposes routing shutdown signals into the validator, draining in-flight work and keymanager requests, tracking detached publishes, and documenting behavior in docs/spec_deviations.md and the README. Done means the listed signal and drain tests pass, shutdown logs appear, and the client exits cleanly.
由索引模型根據 Issue 內容生成。
描述
ethlambda validator installs no signal handler. SIGINT and SIGTERM end it wherever it stands, and in the Docker image, where it runs as PID 1, both are ignored until Docker sends SIGKILL. #799 gave the node graceful SIGINT and SIGTERM handling and left the validator client unchanged, as its description says.
Code references are at 5d8628e9 (current main, with #799 merged).
What happens today
validator::runbuilds a runtime and awaitsethlambda_validator::run(bin/ethlambda/src/validator.rs:245-276). Neither listens for a signal. #799'sShutdownSignals(bin/ethlambda/src/shutdown.rs:31) is used only by the node'swait_for_shutdownand the Hive test driver (bin/ethlambda/src/main.rs:1040,:1163).- With no handler, both signals keep their default action and the process ends at once. The image's entrypoint is the binary itself (
Dockerfile:96), so in a container it is PID 1, where the kernel drops a signal that has no handler.docker stopthen waits its timeout (10 s by default) and sends SIGKILL. - The client already has a stop path.
run_until_stoppedcancels the keymanager server and gives requests in flightKEYMANAGER_DRAIN(60 s) to finish (crates/validator/src/lib.rs:561-627). It runs only when the duty loop returns an error or the slashing-protection record is poisoned. No signal reaches it, andrunreturnsErron every path (lib.rs:558).
What an abrupt kill interrupts
Nothing it interrupts can get a validator slashed:
- Slashing-protection writes go to a
.tmpsibling, are flushed, renamed over the record, and the directory is flushed (crates/validator/src/secure_fs.rs:145-183, called fromcrates/validator/src/slashing_protection/file.rs:266-290). A kill leaves the old file or the new one, and a leftover.tmpis removed by the next write (secure_fs.rs:158-162). Every block and attestation is recorded before it is signed.validator_definitions.ymlis written the same way. - The record lock is an
flockon the validators directory (slashing_protection/file.rs:294-314). The kernel releases it when the process exits, SIGKILL included, so a restart is never refused for a stale lock.
What it does cost:
- A keymanager import cut between saving its keys and registering them. That leaves keys on disk with no slashing-protection record, and the next start refuses the whole client until those keys get a record (
ethlambda slashing-protection import, or the dangerous--init-slashing-protection) (crates/validator/src/http_api/mod.rs:211-219,crates/validator/src/http_api/keystores.rs:406-413). The drain inrun_until_stoppedexists to prevent exactly this, and a signal skips it. - A signed block whose publish is still in flight. The proposal is lost, and the restarted client cannot sign that slot again, because the record already holds it.
- A blinded block's publish runs on a detached task (
crates/validator/src/proposal.rs:755).docs/spec_deviations.md:806-808already notes that a SIGINT in that window ends it. - At gloas, a kill between the block's publish and its envelope's withholds the slot's payload (
proposal.rs:643).
- A blinded block's publish runs on a detached task (
- Attestations, aggregates and sync committee messages signed and not yet submitted are lost, for the same reason.
- Detached tasks are dropped mid-request: proposer preferences (
lib.rs:470), subnet subscriptions (lib.rs:492), builder registrations (lib.rs:512,:543) and the fee-recipient re-send (lib.rs:1191). Each is re-sent later, so this costs at most a slot or an epoch. - No shutdown line in the log, so a clean stop and a crash look the same. Every
docker stoptakes the full timeout.
What other clients do
| Client | Behavior | Read at |
|---|---|---|
| Lighthouse | One handler for the beacon node and the validator client: SIGTERM, SIGINT and SIGHUP all start a shutdown (lighthouse/environment/src/lib.rs:365-388) |
unstable @ 11813cfb |
| Prysm | SIGINT or SIGTERM starts a "duty-aware" shutdown. It waits until the current slot's rewarded duties (attestation, sync committee message, proposal) are done, and through the next slot's too when that slot has one and less than 3 s would be left to restart. It gives up after 8 s, under Docker's default 10 s. A second signal stops at once, and ten more panic (validator/node/node.go:131-147, :156-172, validator/client/duty_aware_shutdown.go:14-27, :137) |
develop @ 0c1e3d24 |
Proposal
- Listen in
validator::runwith #799'sShutdownSignals. It ispub(crate)in the same bin crate, sovalidator.rscan use it as it is. Passethlambda_validator::runa shutdown future or aCancellationToken; the library crate should not install handlers of its own. - Add a third arm to
run_until_stopped'sselect!for that signal. On it:- Stop the duty loop from starting another slot.
- Let the work in flight finish, bounded: the proposal (its own deadline is the attester offset), the gloas envelope, a blinded block's publish, and the slot's attestations.
- Cancel the keymanager server and drain it, as today.
- Log
Shutdown signal receivedwith the signal, thenShutdown complete, and returnOk(())so the process exits 0.
- Escalate on repeats with
ShutdownSignals::force_exit_on_repeats, so a stuck drain can still be ended by hand. - Track the detached tasks that matter, at least the blinded publish, in a
tokio_util::task::TaskTracker, so the shutdown can wait for them. - Docs. Drop the "handles no signals" sentence in
docs/spec_deviations.md:806-808, and say in the README's validator client section how the client stops under systemd and Docker.
To decide
- Drop the duties in flight, or wait. Dropping them at once is safe, since everything signed was recorded first. Waiting a few seconds saves a proposal or a slot's attestations. Prysm waits for the slot's rewarded duties, capped at 8 s.
KEYMANAGER_DRAINis 60 s, against Docker's 10 s default, Kubernetes' 30 s grace period and systemd's 90 s. Either document a longer stop timeout for the client, or shorten the drain when a signal started it. A cut-short import is the one costly outcome above, so shortening it trades one problem for another.- When to start listening. #799 listens from the node's ready line, and a signal during startup keeps its default action. The client's startup decrypts every keystore, which can take a while with many keys (#797), so listening from the top of
runwould need startup to check for a signal as it goes.
Tests
- Like #799's
bin/ethlambda/tests/shutdown.rs: spawnethlambda validatorwith an empty validators directory against a stub HTTP server answering/eth/v1/beacon/genesis,/eth/v1/config/specand/eth/v1/node/syncing. Wait forValidator client starting, send SIGTERM, and assert exit code 0 and both shutdown lines. - Next to the existing
run_until_stoppedtests incrates/validator/src/lib.rs: a shutdown future that resolves while a keymanager request is in flight. The request finishes beforerun_until_stoppedreturns, and the result is a clean stop.
- 主要語言
- Rust
- 星號
- 86
- 分支
- 34
- 平均合併
- 9 天 20 小時
- 30 天內合併 PR
- 77
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
lambdaclass/ethlambda 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 76/100
lambdaclass/ethlambda#850 ·
維護者通常 6 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
lambdaclass/ethlambda#777 · 1 個 reaction ·
維護者通常 6 天內回覆
-
Run all gossip validation tests可能重新可做 @dicethedev 於 9 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 62/100
lambdaclass/ethlambda#725 · 2 則留言 ·
維護者通常 6 天內回覆
-
Handle malformed DB data可能重新可做 @dicethedev 於 115 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 76/100
lambdaclass/ethlambda#529 · 4 則留言 ·
維護者通常 6 天內回覆
-
Document architectural choices可能重新可做 @dicethedev 於 115 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 68/100
lambdaclass/ethlambda#521 · 2 則留言 ·
維護者通常 6 天內回覆
查看 lambdaclass/ethlambda 的全部 Issue
相似的 Issue
-
難度 2/5 1-3 小時 新手友好度 68/100
-
難度 1/5 1 小時以內 新手友好度 82/100
stackabletech/airflow-operator#898 ·
維護者通常 1 天內回覆
-
2 advisory ignore entries no longer match anything (`bytes`, `time` already patched in `Cargo.lock`)可能已有人在做 @zyl71 今天認領。 未關閉
難度 1/5 1 小時以內 新手友好度 88/100
-
難度 1/5 1 小時以內 新手友好度 85/100
stackabletech/druid-operator#905 ·
維護者通常 1 天內回覆
-
難度 1/5 1 小時以內 新手友好度 88/100
bytecodealliance/wrpc#1785 ·
維護者通常 1 天內回覆