validator: shut down gracefully on SIGINT and SIGTERM
Les mainteneurs répondent en général sous 6 jours
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Accessibilité débutants
- 38/100
Piste de recherche
Start with bin/ethlambda/src/validator.rs:245-276 and bin/ethlambda/src/shutdown.rs:31, then trace crates/validator/src/lib.rs:561-627 and its run_until_stopped tests. The issue proposes routing shutdown signals into the validator, draining in-flight work and keymanager requests, tracking detached publishes, and documenting behavior in docs/spec_deviations.md and the README. Done means the listed signal and drain tests pass, shutdown logs appear, and the client exits cleanly.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
ethlambda validator installs no signal handler. SIGINT and SIGTERM end it wherever it stands, and in the Docker image, where it runs as PID 1, both are ignored until Docker sends SIGKILL. #799 gave the node graceful SIGINT and SIGTERM handling and left the validator client unchanged, as its description says.
Code references are at 5d8628e9 (current main, with #799 merged).
What happens today
validator::runbuilds a runtime and awaitsethlambda_validator::run(bin/ethlambda/src/validator.rs:245-276). Neither listens for a signal. #799'sShutdownSignals(bin/ethlambda/src/shutdown.rs:31) is used only by the node'swait_for_shutdownand the Hive test driver (bin/ethlambda/src/main.rs:1040,:1163).- With no handler, both signals keep their default action and the process ends at once. The image's entrypoint is the binary itself (
Dockerfile:96), so in a container it is PID 1, where the kernel drops a signal that has no handler.docker stopthen waits its timeout (10 s by default) and sends SIGKILL. - The client already has a stop path.
run_until_stoppedcancels the keymanager server and gives requests in flightKEYMANAGER_DRAIN(60 s) to finish (crates/validator/src/lib.rs:561-627). It runs only when the duty loop returns an error or the slashing-protection record is poisoned. No signal reaches it, andrunreturnsErron every path (lib.rs:558).
What an abrupt kill interrupts
Nothing it interrupts can get a validator slashed:
- Slashing-protection writes go to a
.tmpsibling, are flushed, renamed over the record, and the directory is flushed (crates/validator/src/secure_fs.rs:145-183, called fromcrates/validator/src/slashing_protection/file.rs:266-290). A kill leaves the old file or the new one, and a leftover.tmpis removed by the next write (secure_fs.rs:158-162). Every block and attestation is recorded before it is signed.validator_definitions.ymlis written the same way. - The record lock is an
flockon the validators directory (slashing_protection/file.rs:294-314). The kernel releases it when the process exits, SIGKILL included, so a restart is never refused for a stale lock.
What it does cost:
- A keymanager import cut between saving its keys and registering them. That leaves keys on disk with no slashing-protection record, and the next start refuses the whole client until those keys get a record (
ethlambda slashing-protection import, or the dangerous--init-slashing-protection) (crates/validator/src/http_api/mod.rs:211-219,crates/validator/src/http_api/keystores.rs:406-413). The drain inrun_until_stoppedexists to prevent exactly this, and a signal skips it. - A signed block whose publish is still in flight. The proposal is lost, and the restarted client cannot sign that slot again, because the record already holds it.
- A blinded block's publish runs on a detached task (
crates/validator/src/proposal.rs:755).docs/spec_deviations.md:806-808already notes that a SIGINT in that window ends it. - At gloas, a kill between the block's publish and its envelope's withholds the slot's payload (
proposal.rs:643).
- A blinded block's publish runs on a detached task (
- Attestations, aggregates and sync committee messages signed and not yet submitted are lost, for the same reason.
- Detached tasks are dropped mid-request: proposer preferences (
lib.rs:470), subnet subscriptions (lib.rs:492), builder registrations (lib.rs:512,:543) and the fee-recipient re-send (lib.rs:1191). Each is re-sent later, so this costs at most a slot or an epoch. - No shutdown line in the log, so a clean stop and a crash look the same. Every
docker stoptakes the full timeout.
What other clients do
| Client | Behavior | Read at |
|---|---|---|
| Lighthouse | One handler for the beacon node and the validator client: SIGTERM, SIGINT and SIGHUP all start a shutdown (lighthouse/environment/src/lib.rs:365-388) |
unstable @ 11813cfb |
| Prysm | SIGINT or SIGTERM starts a "duty-aware" shutdown. It waits until the current slot's rewarded duties (attestation, sync committee message, proposal) are done, and through the next slot's too when that slot has one and less than 3 s would be left to restart. It gives up after 8 s, under Docker's default 10 s. A second signal stops at once, and ten more panic (validator/node/node.go:131-147, :156-172, validator/client/duty_aware_shutdown.go:14-27, :137) |
develop @ 0c1e3d24 |
Proposal
- Listen in
validator::runwith #799'sShutdownSignals. It ispub(crate)in the same bin crate, sovalidator.rscan use it as it is. Passethlambda_validator::runa shutdown future or aCancellationToken; the library crate should not install handlers of its own. - Add a third arm to
run_until_stopped'sselect!for that signal. On it:- Stop the duty loop from starting another slot.
- Let the work in flight finish, bounded: the proposal (its own deadline is the attester offset), the gloas envelope, a blinded block's publish, and the slot's attestations.
- Cancel the keymanager server and drain it, as today.
- Log
Shutdown signal receivedwith the signal, thenShutdown complete, and returnOk(())so the process exits 0.
- Escalate on repeats with
ShutdownSignals::force_exit_on_repeats, so a stuck drain can still be ended by hand. - Track the detached tasks that matter, at least the blinded publish, in a
tokio_util::task::TaskTracker, so the shutdown can wait for them. - Docs. Drop the "handles no signals" sentence in
docs/spec_deviations.md:806-808, and say in the README's validator client section how the client stops under systemd and Docker.
To decide
- Drop the duties in flight, or wait. Dropping them at once is safe, since everything signed was recorded first. Waiting a few seconds saves a proposal or a slot's attestations. Prysm waits for the slot's rewarded duties, capped at 8 s.
KEYMANAGER_DRAINis 60 s, against Docker's 10 s default, Kubernetes' 30 s grace period and systemd's 90 s. Either document a longer stop timeout for the client, or shorten the drain when a signal started it. A cut-short import is the one costly outcome above, so shortening it trades one problem for another.- When to start listening. #799 listens from the node's ready line, and a signal during startup keeps its default action. The client's startup decrypts every keystore, which can take a while with many keys (#797), so listening from the top of
runwould need startup to check for a signal as it goes.
Tests
- Like #799's
bin/ethlambda/tests/shutdown.rs: spawnethlambda validatorwith an empty validators directory against a stub HTTP server answering/eth/v1/beacon/genesis,/eth/v1/config/specand/eth/v1/node/syncing. Wait forValidator client starting, send SIGTERM, and assert exit code 0 and both shutdown lines. - Next to the existing
run_until_stoppedtests incrates/validator/src/lib.rs: a shutdown future that resolves while a keymanager request is in flight. The request finishes beforerun_until_stoppedreturns, and the result is a clean stop.
- Langage dominant
- Rust
- Étoiles
- 86
- Forks
- 34
- Merge moyen
- 9 j 20 h
- PR mergées (30 j)
- 77
Préparer son environnement
- Fournit un Dockerfile ou un fichier Docker Compose
- Propose un modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de lambdaclass/ethlambda
-
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
lambdaclass/ethlambda#850 ·
Les mainteneurs répondent en général sous 6 jours
-
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
lambdaclass/ethlambda#777 · 1 réaction ·
Les mainteneurs répondent en général sous 6 jours
-
Run all gossip validation testsPeut-être à nouveau libre @dicethedev l’a pris il y a 9 jours, et aucune pull request n’est ouverte. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 62/100
lambdaclass/ethlambda#725 · 2 commentaires ·
Les mainteneurs répondent en général sous 6 jours
-
Handle malformed DB dataPeut-être à nouveau libre @dicethedev l’a pris il y a 115 jours, et aucune pull request n’est ouverte. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
lambdaclass/ethlambda#529 · 4 commentaires ·
Les mainteneurs répondent en général sous 6 jours
-
Document architectural choicesPeut-être à nouveau libre @dicethedev l’a pris il y a 115 jours, et aucune pull request n’est ouverte. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
lambdaclass/ethlambda#521 · 2 commentaires ·
Les mainteneurs répondent en général sous 6 jours
Toutes les issues de lambdaclass/ethlambda
Issues similaires
-
Difficulté 1/5 Moins d'une heure Accessibilité débutants 88/100
bytecodealliance/wrpc#1785 ·
Les mainteneurs répondent en général sous 1 jour
-
2 advisory ignore entries no longer match anything (`quick-xml` already patched in `Cargo.lock`)Ouverte
Difficulté 1/5 Moins d'une heure Accessibilité débutants 85/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
longbridge/nsh#2758 · 1 commentaire ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 84/100
antithesishq/bombadil#473 ·
Les mainteneurs répondent en général sous 2 jours
-
bench/api.sh exec test fails with curl exit 52, and --ignore-failure still records the timingsOuvertebug
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
libracoder/kiri#669 ·
Les mainteneurs répondent en général sous 1 jour