[Bug] In-flight disk scan falsely fails health clockConsistent
まだ誰も着手していません。
評価
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 初心者へのやさしさ
- 78/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- node.js, typescript
調査の方向性
Read src/health/service.ts and src/health/state.ts to follow checkpoint timing and the future timestamp check, then inspect src/storage/metrics.ts and the existing health tests. Reproduce the race by holding attestation until storage metrics refresh, and verify that a scan completed during the wait is deferred to the next checkpoint while a genuinely future timestamp still fails. Done means the regression passes without widening freshness windows.
索引モデルが issue の本文から書いたものです。
説明
Summary
runHealthCheckpoint records now before awaiting peer attestation, then reads storage metrics after that await. A disk scan that finishes in between stores updatedAt slightly ahead of now. evaluateHealth treats that as a clock rollback, refuses the checkpoint, and GET /api/node/health stays degraded until the next successful round. Replication, peer attestations, and the host clock can all be healthy. Widening freshness windows, or debouncing a monitor that alerts on state != ready, does not fix it.
Steps to reproduce
- Run ipfs-node v0.1.0 with more than one configured node, so
health.requiredPeerCountis at least 1 (the default). Leave the health intervals anddiskUsageScanPeriodat their defaults: a checkpoint every 60 seconds, a disk scan every 30 seconds. - Let the storage scan run long enough that it sometimes completes while
countAttestationsis in flight. A blockstore directory walk of several seconds is enough when the process timer phase sits near scan completion. - Poll
GET /api/node/healthacross a few checkpoint intervals.
A deterministic test can hold attestation until refreshStorageMetrics has set updatedAt, then release it.
Expected behavior
A storage measurement taken during the checkpoint attempt is either included with an evaluation time that is not earlier than updatedAt, or left for the next round. checks.clockConsistent stays true unless the clock actually moved behind a timestamp this process already recorded. With a fresh scan, a successful repair cycle, and the required peer attestations, state stays ready.
Actual behavior
The checkpoint fails closed. Until the next successful round (up to health.checkpointIntervalMs):
stateisdegradedwhilecheckpoint.ageMsis still belowcheckpoint.maxAgeMsstorage.measuredAtis a few milliseconds afterevaluatedAt, whilestorage.measurementAgeMsat read time is still well insidehealth.storageMaxAgeMschecks.clockConsistentis false. Read-time refresh will not set it back to true, so later polls keep the failure even after wall time has moved paststorage.measuredAtchecks.checkpointFresh,checks.storageFresh, andchecks.repairFreshare false with it, because refresh requiresclockConsistentfor all threereplication.repairRequiredstays true. That flag means repair is part of the readiness contract (replication.enabledandreplication.repairEnabled), not that the last cycle failed. A successful cycle with backlog 0 still reportsrepairRequired: truechecks.helia,checks.startupReconciliation,checks.storageReserve, andchecks.peerAttestationsstay truetimestampminusevaluatedAtgrows through the interval. That gap is the age of the cached snapshot, including whilestateisready
The next checkpoint that does not overlap a scan completion returns ready. Each overlap therefore produces about a minute of degraded, and the process timer keeps the same phase for its whole lifetime, so the flap repeats until the phase no longer meets scan completion. A restart only draws a new phase.
Environment
ipfs-node v0.1.0, current health checkpoint code. Defaults: health.checkpointIntervalMs 60000, health.maxCheckpointAgeMs 180000, health.storageMaxAgeMs 120000, health.repairMaxAgeMs 3600000, diskUsageScanPeriod */30 * * * * *, repair enabled, one required peer. No configuration change is required to hit this.
Cause
In src/health/service.ts, runHealthCheckpoint does this:
const now = Date.now()
const repair = getRepairHealthEvidence()
const attestedPeers = await countAttestations(now)
const storage = getStorageMetrics()
evaluateHealth in src/health/state.ts rejects the round when storage.updatedAt, the repair completion, or the previous checkpoint is after now (notInFuture). Repair evidence is read before the await, so it stays in the past. Storage is read after the await. refreshStorageMetrics in src/storage/metrics.ts assigns updatedAt: Date.now() only when the directory walk finishes, which can be while attestation is waiting.
refreshHealthSnapshot then latches checks.clockConsistent false and folds it into checkpointFresh, storageFresh, and repairFresh. One ordering glitch is reported as four freshness failures. docs/operations/monitoring.md describes clockConsistent: false as the host clock moving backwards, which is the conclusion an operator will draw.
The health timer is setInterval(health.checkpointIntervalMs) from process start, so its phase never moves. The disk scan is a wall-clock cron (src/disk-usage.cron.ts) every 30 seconds. Those two stay aligned for the whole uptime. Scan duration only has to land inside the attestation wait, bounded by health.peerAttestationTimeoutMs (default 5 seconds, often much shorter), to fail the round.
This is separate from #41. There, a real repair backlog stops checkpoints and the node becomes stale. Here the repair cycle can be successful, inside health.repairMaxAgeMs, with backlog 0.
Proposed solution
Read storage metrics in the same synchronous stretch as now, before the attestation await:
const now = Date.now()
const repair = getRepairHealthEvidence()
const storage = getStorageMetrics()
const attestedPeers = await countAttestations(now)
On one JavaScript turn, updatedAt cannot move between Date.now() and getStorageMetrics(). A scan that finishes during attestation is picked up on the next checkpoint. The default storage window is 120 seconds and the scan period is 30 seconds, so skipping one in-flight sample does not fail storageFresh.
Keep the future-timestamp rejection. A tolerance on notInFuture would also accept a real backward clock step. Do not widen maxCheckpointAgeMs, storageMaxAgeMs, or repairMaxAgeMs.
Add a regression in which attestation waits until a storage refresh has set updatedAt, then assert the checkpoint still completes when that scan is otherwise inside storageMaxAgeMs. Keep the existing evaluateHealth test that refuses a storage measurement from ahead of the attempt, so a real rollback still fails closed.
- 主要言語
- TypeScript
- スター
- 309
- フォーク
- 5
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートなし
- コントリビューションガイドなし
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
Adamant-im/ipfs-node のほかの issue
-
Decentralization IPFS Privacy Research Security
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
Adamant-im/ipfs-node#31 ·
-
enhancement Infrastructure IPFS
難易度 5/5 1週間以上 初心者へのやさしさ 38/100
Adamant-im/ipfs-node#30 ·
-
enhancement Infrastructure IPFS
難易度 5/5 1週間以上 初心者へのやさしさ 30/100
Adamant-im/ipfs-node#29 ·
-
Composite task Decentralization Infrastructure IPFS
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
Adamant-im/ipfs-node#28 ·
-
Composite task Decentralization Encryption Governance IPFS
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
Adamant-im/ipfs-node#27 · コメント 1 件 ·
Adamant-im/ipfs-node の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
code-yeongyu/senpi#7139 ·
メンテナーはふだん 1 日以内に返信
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
難易度 1/5 1〜3時間 初心者へのやさしさ 72/100
lingdojo/kana-dojo#34395 · コメント 1 件 · リアクション 5 件 ·
メンテナーはふだん 1 日以内に返信
-
bug status: needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
mastra-ai/mastra#31093 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
area:web p2 product:business triage:active type:bug
難易度 1/5 1時間未満 初心者へのやさしさ 89/100
cuadrao-ai/argus#2328 ·
メンテナーはふだん 1 日以内に返信
-
bug frontend good first issue help wanted
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
nikhilb2/simple_invoicing#907 ·