validator: shut down gracefully on SIGINT and SIGTERM

未关闭
#835 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 6 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
38/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
docker, rust
领域
backend, devops

调研方向

Start with bin/ethlambda/src/validator.rs:245-276 and bin/ethlambda/src/shutdown.rs:31, then trace crates/validator/src/lib.rs:561-627 and its run_until_stopped tests. The issue proposes routing shutdown signals into the validator, draining in-flight work and keymanager requests, tracking detached publishes, and documenting behavior in docs/spec_deviations.md and the README. Done means the listed signal and drain tests pass, shutdown logs appear, and the client exits cleanly.

由索引模型根据 Issue 内容生成。

描述

ethlambda validator installs no signal handler. SIGINT and SIGTERM end it wherever it stands, and in the Docker image, where it runs as PID 1, both are ignored until Docker sends SIGKILL. #799 gave the node graceful SIGINT and SIGTERM handling and left the validator client unchanged, as its description says.

Code references are at 5d8628e9 (current main, with #799 merged).

What happens today

  • validator::run builds a runtime and awaits ethlambda_validator::run (bin/ethlambda/src/validator.rs:245-276). Neither listens for a signal. #799's ShutdownSignals (bin/ethlambda/src/shutdown.rs:31) is used only by the node's wait_for_shutdown and the Hive test driver (bin/ethlambda/src/main.rs:1040, :1163).
  • With no handler, both signals keep their default action and the process ends at once. The image's entrypoint is the binary itself (Dockerfile:96), so in a container it is PID 1, where the kernel drops a signal that has no handler. docker stop then waits its timeout (10 s by default) and sends SIGKILL.
  • The client already has a stop path. run_until_stopped cancels the keymanager server and gives requests in flight KEYMANAGER_DRAIN (60 s) to finish (crates/validator/src/lib.rs:561-627). It runs only when the duty loop returns an error or the slashing-protection record is poisoned. No signal reaches it, and run returns Err on every path (lib.rs:558).

What an abrupt kill interrupts

Nothing it interrupts can get a validator slashed:

  • Slashing-protection writes go to a .tmp sibling, are flushed, renamed over the record, and the directory is flushed (crates/validator/src/secure_fs.rs:145-183, called from crates/validator/src/slashing_protection/file.rs:266-290). A kill leaves the old file or the new one, and a leftover .tmp is removed by the next write (secure_fs.rs:158-162). Every block and attestation is recorded before it is signed. validator_definitions.yml is written the same way.
  • The record lock is an flock on the validators directory (slashing_protection/file.rs:294-314). The kernel releases it when the process exits, SIGKILL included, so a restart is never refused for a stale lock.

What it does cost:

  • A keymanager import cut between saving its keys and registering them. That leaves keys on disk with no slashing-protection record, and the next start refuses the whole client until those keys get a record (ethlambda slashing-protection import, or the dangerous --init-slashing-protection) (crates/validator/src/http_api/mod.rs:211-219, crates/validator/src/http_api/keystores.rs:406-413). The drain in run_until_stopped exists to prevent exactly this, and a signal skips it.
  • A signed block whose publish is still in flight. The proposal is lost, and the restarted client cannot sign that slot again, because the record already holds it.
    • A blinded block's publish runs on a detached task (crates/validator/src/proposal.rs:755). docs/spec_deviations.md:806-808 already notes that a SIGINT in that window ends it.
    • At gloas, a kill between the block's publish and its envelope's withholds the slot's payload (proposal.rs:643).
  • Attestations, aggregates and sync committee messages signed and not yet submitted are lost, for the same reason.
  • Detached tasks are dropped mid-request: proposer preferences (lib.rs:470), subnet subscriptions (lib.rs:492), builder registrations (lib.rs:512, :543) and the fee-recipient re-send (lib.rs:1191). Each is re-sent later, so this costs at most a slot or an epoch.
  • No shutdown line in the log, so a clean stop and a crash look the same. Every docker stop takes the full timeout.

What other clients do

Client Behavior Read at
Lighthouse One handler for the beacon node and the validator client: SIGTERM, SIGINT and SIGHUP all start a shutdown (lighthouse/environment/src/lib.rs:365-388) unstable @ 11813cfb
Prysm SIGINT or SIGTERM starts a "duty-aware" shutdown. It waits until the current slot's rewarded duties (attestation, sync committee message, proposal) are done, and through the next slot's too when that slot has one and less than 3 s would be left to restart. It gives up after 8 s, under Docker's default 10 s. A second signal stops at once, and ten more panic (validator/node/node.go:131-147, :156-172, validator/client/duty_aware_shutdown.go:14-27, :137) develop @ 0c1e3d24

Proposal

  • Listen in validator::run with #799's ShutdownSignals. It is pub(crate) in the same bin crate, so validator.rs can use it as it is. Pass ethlambda_validator::run a shutdown future or a CancellationToken; the library crate should not install handlers of its own.
  • Add a third arm to run_until_stopped's select! for that signal. On it:
    1. Stop the duty loop from starting another slot.
    2. Let the work in flight finish, bounded: the proposal (its own deadline is the attester offset), the gloas envelope, a blinded block's publish, and the slot's attestations.
    3. Cancel the keymanager server and drain it, as today.
    4. Log Shutdown signal received with the signal, then Shutdown complete, and return Ok(()) so the process exits 0.
  • Escalate on repeats with ShutdownSignals::force_exit_on_repeats, so a stuck drain can still be ended by hand.
  • Track the detached tasks that matter, at least the blinded publish, in a tokio_util::task::TaskTracker, so the shutdown can wait for them.
  • Docs. Drop the "handles no signals" sentence in docs/spec_deviations.md:806-808, and say in the README's validator client section how the client stops under systemd and Docker.

To decide

  • Drop the duties in flight, or wait. Dropping them at once is safe, since everything signed was recorded first. Waiting a few seconds saves a proposal or a slot's attestations. Prysm waits for the slot's rewarded duties, capped at 8 s.
  • KEYMANAGER_DRAIN is 60 s, against Docker's 10 s default, Kubernetes' 30 s grace period and systemd's 90 s. Either document a longer stop timeout for the client, or shorten the drain when a signal started it. A cut-short import is the one costly outcome above, so shortening it trades one problem for another.
  • When to start listening. #799 listens from the node's ready line, and a signal during startup keeps its default action. The client's startup decrypts every keystore, which can take a while with many keys (#797), so listening from the top of run would need startup to check for a signal as it goes.

Tests

  • Like #799's bin/ethlambda/tests/shutdown.rs: spawn ethlambda validator with an empty validators directory against a stub HTTP server answering /eth/v1/beacon/genesis, /eth/v1/config/spec and /eth/v1/node/syncing. Wait for Validator client starting, send SIGTERM, and assert exit code 0 and both shutdown lines.
  • Next to the existing run_until_stopped tests in crates/validator/src/lib.rs: a shutdown future that resolves while a keymanager request is in flight. The request finishes before run_until_stopped returns, and the result is a clean stop.
主要语言
Rust
星标
86
派生
34
平均合并
9 天 20 小时
30 天内合并 PR
77

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

lambdaclass/ethlambda 的其他 Issue

查看 lambdaclass/ethlambda 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。