validator: warn when the local clock disagrees with the beacon node
維護者通常 6 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 35/100
- Issue 類型
- 功能
- 描述清晰度
- 基本清楚
- 活躍度
- 活躍
- 技術堆疊
- rust
- 領域
- backend, observability
研究方向
Start with crates/validator/src/slot_clock.rs, crates/validator/src/lib.rs, and the beacon node syncing DTO and HTTP client in crates/validator/src/beacon_node/; read how epoch refresh and slot timing currently work. The issue proposes clock offset checks, step warnings, and metrics, but leaves key design choices unresolved, including the warning threshold and whether to use an external time source. Done means those decisions are settled and the proposed checks and tests cover the selected behavior.
由索引模型根據 Issue 內容生成。
描述
The validator client times every duty off the local wall clock and never checks that clock against anything. A clock that is off by a second or more makes duties early or late, and nothing the client logs or exports says why.
Code references are at bf66e86c.
What happens today
SlotClockis seeded once from the node's genesis time and/eth/v1/config/spec, then readsSystemTime::now()for the rest of the process (crates/validator/src/slot_clock.rs:1-6,slot_atat:96-101, built inrunatcrates/validator/src/lib.rs:253). The duty loop wakes on it at every slot boundary (lib.rs:381-382).GET /eth/v1/node/syncingis already called once an epoch (refresh_epoch,lib.rs:979).SyncingDtodeclares onlyis_syncing,is_optimisticandel_offline(crates/validator/src/beacon_node/dto.rs:137-156), so the node'shead_slotandsync_distancearrive and are dropped.HttpBeaconNode::getdecodes the body and discards the response headers,Dateincluded (crates/validator/src/beacon_node/http.rs:192-206).- A wall-clock step is absorbed without a log line. After a forward step
next_slot_to_serveserves the current slot at once, and after a backward step it moves on toserved + 1(slot_clock.rs:328-338). A clock stepped back by ten minutes makes the loop sleep ten minutes. ethlambda_validator_publication_delay_secondsis measured against the client's own clock (lib.rs:756-759), so it reads normal however far that clock is off.
Why it matters
Against an ethlambda beacon node:
- Clock ahead by more than about 500 ms: proposals are refused. The client asks for its block at its own slot boundary, after signing the RANDAO reveal (and, on an epoch's first slot, after the epoch refresh). The node refuses to produce a block for a slot that has not started by its clock, with
MAXIMUM_GOSSIP_CLOCK_DISPARITY(500 ms) of allowance:refuse_future_slot(crates/net/rpc/src/beacon/proposal.rs:465), called byproduceBlockV3(proposal.rs:552) andproduceBlockV4(crates/net/rpc/src/beacon/gloas_proposal.rs:264). The client never retries a proposal within its slot (crates/validator/src/proposal.rs:93-100), so the slot is lost. The log saysFailed to propose this slot's blockwith a 400 (lib.rs:938), and nothing points at the clock. - Ahead by a few seconds: attestations vote for the parent. They go out before the slot's block has arrived. Past the attester offset plus 500 ms (about 4.5 s before gloas, 3.5 s at gloas) the node refuses them as from the future (
crates/net/rpc/src/beacon/pool.rs:175-182), as every peer's gossip validation would. - Clock behind: nothing is refused. Blocks go out late, lose proposer boost and risk a late-block reorg. Attestations go out late and miss the head. Every series this client exports looks healthy. This is the direction an operator is least likely to notice.
When the client and its beacon node share a host they share the clock, so a comparison between the two only catches a split setup (separate machines or VMs). Catching a skewed host needs a reference outside it.
What other clients do
| Client | Behavior | Read at |
|---|---|---|
| Lighthouse | Marks a beacon node unhealthy (CandidateError::TimeDiscrepancy) when the node's head_slot is more than one slot past the client's wall slot, and warns A connected beacon node errored during routine health check (validator_client/beacon_node_fallback/src/lib.rs:53, :314-318, :655-660). That catches a client clock behind by a slot or more. A clock that runs ahead only makes the node look further behind, which feeds node ranking (:319) |
unstable @ 11813cfb |
| Nimbus | The validator client used to measure its offset from the beacon node through Nimbus's own POST /nimbus/v1/timesync. status-im/nimbus-eth2#7406 ("VC: Remove timeoffset check function", 91581318, August 2025) removed the check without a stated reason. The node still serves the endpoint (beacon_chain/rpc/rest_nimbus_api.nim:454). The client does shut down when the wall clock moves back by more than one slot while it sleeps (beacon_chain/validator_client/common.nim:1511-1520) |
unstable @ 9bed158a |
Proposal
Two checks on calls the client already makes, plus a log line for clock steps:
- Slot-level comparison from
/node/syncing. Declarehead_slotandsync_distanceinSyncingDto.head_slotmore than one slot past the client's wall slot means the client's clock is behind (Lighthouse's check).head_slot + sync_distanceis the node's own wall slot whenever its head is not ahead of its clock. ethlambda's node computessync_distanceexactly that way (crates/net/rpc/src/beacon/node.rs:62-63). A difference from the client's wall slot, sampled on both sides of the request so a slot boundary inside the round trip is not counted, means the two clocks disagree by most of a slot or more.
- Second-level comparison from
Date. Read theDateheader ingetandpostand estimate the offset asDateminus the midpoint of send and receive. The Beacon API does not require the header, so a node that omits it gets only the slot-level check. The header has one-second resolution, so the estimate is good to about ±0.5 s plus half the round trip, which is enough to flag a skew of 2 s or more after a few samples. - Clock steps. In the duty loop, warn when the wall clock lands before the slot just served (a backward step) or more than one slot past it (a forward step, a suspended process or a paused VM).
- Surface it. A warning that names the offset and the node, and a gauge (
ethlambda_validator_beacon_node_clock_offset_seconds, plus a counter of detected steps), so it can be alerted on.
To decide
- Resolution. About 500 ms already gets proposals refused by an ethlambda node, and that is below what
Date(1 s) or a slot comparison can see. Is a 1-2 s warning enough, or is a finer source worth it: a timesync-style endpoint on ethlambda's own node, as Nimbus had, or an NTP query? - Which side is wrong. A client-vs-node difference does not say which clock is off. With several
--beacon-nodes, the majority can. - Warn only, or refuse. A warning seems right, since the node's clock may be the wrong one. Marking a node unhealthy for failover, as Lighthouse does, is a further step.
- A shared clock. Catching a skewed host where both processes run needs a signal from the chain, for example
headevents arriving consistently before the slot starts or several seconds after it. The client deliberately does not drive duties from the event stream (slot_clock.rs:3-6); reading it for this diagnostic alone would be new. - The beacon node side. A node with a skewed clock sees every gossip block as early or late. A node-side check is a separate issue.
Tests
MockBeaconNodeanswering/node/syncingwith ahead_slottwo slots past the client's clock: the refresh logs the discrepancy and sets the gauge. One slot past: no warning.- An offset estimator fed responses with a
Dateheader two seconds ahead and behind. next_slot_to_serve(or the loop around it) reporting a backward step and a multi-slot forward step.
- 主要語言
- Rust
- 星號
- 86
- 分支
- 34
- 平均合併
- 9 天 20 小時
- 30 天內合併 PR
- 77
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
lambdaclass/ethlambda 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 76/100
lambdaclass/ethlambda#850 ·
維護者通常 6 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
lambdaclass/ethlambda#777 · 1 個 reaction ·
維護者通常 6 天內回覆
-
Run all gossip validation tests可能重新可做 @dicethedev 於 9 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 62/100
lambdaclass/ethlambda#725 · 2 則留言 ·
維護者通常 6 天內回覆
-
Handle malformed DB data可能重新可做 @dicethedev 於 115 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 76/100
lambdaclass/ethlambda#529 · 4 則留言 ·
維護者通常 6 天內回覆
-
Document architectural choices可能重新可做 @dicethedev 於 115 天前認領,目前沒有進行中的 PR。 未關閉
難度 2/5 1-3 小時 新手友好度 68/100
lambdaclass/ethlambda#521 · 2 則留言 ·
維護者通常 6 天內回覆
查看 lambdaclass/ethlambda 的全部 Issue
相似的 Issue
-
難度 1/5 1 小時以內 新手友好度 88/100
bytecodealliance/wrpc#1785 ·
維護者通常 1 天內回覆
-
2 advisory ignore entries no longer match anything (`quick-xml` already patched in `Cargo.lock`)未關閉
難度 1/5 1 小時以內 新手友好度 85/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
longbridge/nsh#2758 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 84/100
antithesishq/bombadil#473 ·
維護者通常 2 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 72/100
libracoder/kiri#669 ·
維護者通常 1 天內回覆