validator: warn when the local clock disagrees with the beacon node
Los mantenedores suelen responder en 6 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 35/100
- Tipo de issue
- Nueva funcionalidad
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- rust
- Área
- backend, observability
Línea de trabajo
Start with crates/validator/src/slot_clock.rs, crates/validator/src/lib.rs, and the beacon node syncing DTO and HTTP client in crates/validator/src/beacon_node/; read how epoch refresh and slot timing currently work. The issue proposes clock offset checks, step warnings, and metrics, but leaves key design choices unresolved, including the warning threshold and whether to use an external time source. Done means those decisions are settled and the proposed checks and tests cover the selected behavior.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
The validator client times every duty off the local wall clock and never checks that clock against anything. A clock that is off by a second or more makes duties early or late, and nothing the client logs or exports says why.
Code references are at bf66e86c.
What happens today
SlotClockis seeded once from the node's genesis time and/eth/v1/config/spec, then readsSystemTime::now()for the rest of the process (crates/validator/src/slot_clock.rs:1-6,slot_atat:96-101, built inrunatcrates/validator/src/lib.rs:253). The duty loop wakes on it at every slot boundary (lib.rs:381-382).GET /eth/v1/node/syncingis already called once an epoch (refresh_epoch,lib.rs:979).SyncingDtodeclares onlyis_syncing,is_optimisticandel_offline(crates/validator/src/beacon_node/dto.rs:137-156), so the node'shead_slotandsync_distancearrive and are dropped.HttpBeaconNode::getdecodes the body and discards the response headers,Dateincluded (crates/validator/src/beacon_node/http.rs:192-206).- A wall-clock step is absorbed without a log line. After a forward step
next_slot_to_serveserves the current slot at once, and after a backward step it moves on toserved + 1(slot_clock.rs:328-338). A clock stepped back by ten minutes makes the loop sleep ten minutes. ethlambda_validator_publication_delay_secondsis measured against the client's own clock (lib.rs:756-759), so it reads normal however far that clock is off.
Why it matters
Against an ethlambda beacon node:
- Clock ahead by more than about 500 ms: proposals are refused. The client asks for its block at its own slot boundary, after signing the RANDAO reveal (and, on an epoch's first slot, after the epoch refresh). The node refuses to produce a block for a slot that has not started by its clock, with
MAXIMUM_GOSSIP_CLOCK_DISPARITY(500 ms) of allowance:refuse_future_slot(crates/net/rpc/src/beacon/proposal.rs:465), called byproduceBlockV3(proposal.rs:552) andproduceBlockV4(crates/net/rpc/src/beacon/gloas_proposal.rs:264). The client never retries a proposal within its slot (crates/validator/src/proposal.rs:93-100), so the slot is lost. The log saysFailed to propose this slot's blockwith a 400 (lib.rs:938), and nothing points at the clock. - Ahead by a few seconds: attestations vote for the parent. They go out before the slot's block has arrived. Past the attester offset plus 500 ms (about 4.5 s before gloas, 3.5 s at gloas) the node refuses them as from the future (
crates/net/rpc/src/beacon/pool.rs:175-182), as every peer's gossip validation would. - Clock behind: nothing is refused. Blocks go out late, lose proposer boost and risk a late-block reorg. Attestations go out late and miss the head. Every series this client exports looks healthy. This is the direction an operator is least likely to notice.
When the client and its beacon node share a host they share the clock, so a comparison between the two only catches a split setup (separate machines or VMs). Catching a skewed host needs a reference outside it.
What other clients do
| Client | Behavior | Read at |
|---|---|---|
| Lighthouse | Marks a beacon node unhealthy (CandidateError::TimeDiscrepancy) when the node's head_slot is more than one slot past the client's wall slot, and warns A connected beacon node errored during routine health check (validator_client/beacon_node_fallback/src/lib.rs:53, :314-318, :655-660). That catches a client clock behind by a slot or more. A clock that runs ahead only makes the node look further behind, which feeds node ranking (:319) |
unstable @ 11813cfb |
| Nimbus | The validator client used to measure its offset from the beacon node through Nimbus's own POST /nimbus/v1/timesync. status-im/nimbus-eth2#7406 ("VC: Remove timeoffset check function", 91581318, August 2025) removed the check without a stated reason. The node still serves the endpoint (beacon_chain/rpc/rest_nimbus_api.nim:454). The client does shut down when the wall clock moves back by more than one slot while it sleeps (beacon_chain/validator_client/common.nim:1511-1520) |
unstable @ 9bed158a |
Proposal
Two checks on calls the client already makes, plus a log line for clock steps:
- Slot-level comparison from
/node/syncing. Declarehead_slotandsync_distanceinSyncingDto.head_slotmore than one slot past the client's wall slot means the client's clock is behind (Lighthouse's check).head_slot + sync_distanceis the node's own wall slot whenever its head is not ahead of its clock. ethlambda's node computessync_distanceexactly that way (crates/net/rpc/src/beacon/node.rs:62-63). A difference from the client's wall slot, sampled on both sides of the request so a slot boundary inside the round trip is not counted, means the two clocks disagree by most of a slot or more.
- Second-level comparison from
Date. Read theDateheader ingetandpostand estimate the offset asDateminus the midpoint of send and receive. The Beacon API does not require the header, so a node that omits it gets only the slot-level check. The header has one-second resolution, so the estimate is good to about ±0.5 s plus half the round trip, which is enough to flag a skew of 2 s or more after a few samples. - Clock steps. In the duty loop, warn when the wall clock lands before the slot just served (a backward step) or more than one slot past it (a forward step, a suspended process or a paused VM).
- Surface it. A warning that names the offset and the node, and a gauge (
ethlambda_validator_beacon_node_clock_offset_seconds, plus a counter of detected steps), so it can be alerted on.
To decide
- Resolution. About 500 ms already gets proposals refused by an ethlambda node, and that is below what
Date(1 s) or a slot comparison can see. Is a 1-2 s warning enough, or is a finer source worth it: a timesync-style endpoint on ethlambda's own node, as Nimbus had, or an NTP query? - Which side is wrong. A client-vs-node difference does not say which clock is off. With several
--beacon-nodes, the majority can. - Warn only, or refuse. A warning seems right, since the node's clock may be the wrong one. Marking a node unhealthy for failover, as Lighthouse does, is a further step.
- A shared clock. Catching a skewed host where both processes run needs a signal from the chain, for example
headevents arriving consistently before the slot starts or several seconds after it. The client deliberately does not drive duties from the event stream (slot_clock.rs:3-6); reading it for this diagnostic alone would be new. - The beacon node side. A node with a skewed clock sees every gossip block as early or late. A node-side check is a separate issue.
Tests
MockBeaconNodeanswering/node/syncingwith ahead_slottwo slots past the client's clock: the refresh logs the discrepancy and sets the gauge. One slot past: no warning.- An offset estimator fed responses with a
Dateheader two seconds ahead and behind. next_slot_to_serve(or the loop around it) reporting a backward step and a multi-slot forward step.
- Lenguaje dominante
- Rust
- Estrellas
- 86
- Forks
- 34
- Merge medio
- 9 d 20 h
- PR fusionados (30 d)
- 77
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de lambdaclass/ethlambda
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
lambdaclass/ethlambda#850 ·
Los mantenedores suelen responder en 6 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
lambdaclass/ethlambda#777 · 1 reacción ·
Los mantenedores suelen responder en 6 días
-
Run all gossip validation testsQuizá libre de nuevo @dicethedev la tomó hace 9 días y no hay ningún pull request abierto. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
lambdaclass/ethlambda#725 · 2 comentarios ·
Los mantenedores suelen responder en 6 días
-
Handle malformed DB dataQuizá libre de nuevo @dicethedev la tomó hace 115 días y no hay ningún pull request abierto. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
lambdaclass/ethlambda#529 · 4 comentarios ·
Los mantenedores suelen responder en 6 días
-
Document architectural choicesQuizá libre de nuevo @dicethedev la tomó hace 115 días y no hay ningún pull request abierto. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
lambdaclass/ethlambda#521 · 2 comentarios ·
Los mantenedores suelen responder en 6 días
Todos los issues de lambdaclass/ethlambda
Issues similares
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 88/100
bytecodealliance/wrpc#1785 ·
Los mantenedores suelen responder en 1 día
-
2 advisory ignore entries no longer match anything (`quick-xml` already patched in `Cargo.lock`)Abierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
longbridge/nsh#2758 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
antithesishq/bombadil#473 ·
Los mantenedores suelen responder en 2 días
-
bench/api.sh exec test fails with curl exit 52, and --ignore-failure still records the timingsAbiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
libracoder/kiri#669 ·
Los mantenedores suelen responder en 1 día