[Bug] In-flight disk scan falsely fails health clockConsistent

オープン
#49 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
78/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
node.js, typescript

調査の方向性

Read src/health/service.ts and src/health/state.ts to follow checkpoint timing and the future timestamp check, then inspect src/storage/metrics.ts and the existing health tests. Reproduce the race by holding attestation until storage metrics refresh, and verify that a scan completed during the wait is deferred to the next checkpoint while a genuinely future timestamp still fails. Done means the regression passes without widening freshness windows.

索引モデルが issue の本文から書いたものです。

説明

APIs bug NodeJS

Summary

runHealthCheckpoint records now before awaiting peer attestation, then reads storage metrics after that await. A disk scan that finishes in between stores updatedAt slightly ahead of now. evaluateHealth treats that as a clock rollback, refuses the checkpoint, and GET /api/node/health stays degraded until the next successful round. Replication, peer attestations, and the host clock can all be healthy. Widening freshness windows, or debouncing a monitor that alerts on state != ready, does not fix it.

Steps to reproduce

  1. Run ipfs-node v0.1.0 with more than one configured node, so health.requiredPeerCount is at least 1 (the default). Leave the health intervals and diskUsageScanPeriod at their defaults: a checkpoint every 60 seconds, a disk scan every 30 seconds.
  2. Let the storage scan run long enough that it sometimes completes while countAttestations is in flight. A blockstore directory walk of several seconds is enough when the process timer phase sits near scan completion.
  3. Poll GET /api/node/health across a few checkpoint intervals.

A deterministic test can hold attestation until refreshStorageMetrics has set updatedAt, then release it.

Expected behavior

A storage measurement taken during the checkpoint attempt is either included with an evaluation time that is not earlier than updatedAt, or left for the next round. checks.clockConsistent stays true unless the clock actually moved behind a timestamp this process already recorded. With a fresh scan, a successful repair cycle, and the required peer attestations, state stays ready.

Actual behavior

The checkpoint fails closed. Until the next successful round (up to health.checkpointIntervalMs):

  • state is degraded while checkpoint.ageMs is still below checkpoint.maxAgeMs
  • storage.measuredAt is a few milliseconds after evaluatedAt, while storage.measurementAgeMs at read time is still well inside health.storageMaxAgeMs
  • checks.clockConsistent is false. Read-time refresh will not set it back to true, so later polls keep the failure even after wall time has moved past storage.measuredAt
  • checks.checkpointFresh, checks.storageFresh, and checks.repairFresh are false with it, because refresh requires clockConsistent for all three
  • replication.repairRequired stays true. That flag means repair is part of the readiness contract (replication.enabled and replication.repairEnabled), not that the last cycle failed. A successful cycle with backlog 0 still reports repairRequired: true
  • checks.helia, checks.startupReconciliation, checks.storageReserve, and checks.peerAttestations stay true
  • timestamp minus evaluatedAt grows through the interval. That gap is the age of the cached snapshot, including while state is ready

The next checkpoint that does not overlap a scan completion returns ready. Each overlap therefore produces about a minute of degraded, and the process timer keeps the same phase for its whole lifetime, so the flap repeats until the phase no longer meets scan completion. A restart only draws a new phase.

Environment

ipfs-node v0.1.0, current health checkpoint code. Defaults: health.checkpointIntervalMs 60000, health.maxCheckpointAgeMs 180000, health.storageMaxAgeMs 120000, health.repairMaxAgeMs 3600000, diskUsageScanPeriod */30 * * * * *, repair enabled, one required peer. No configuration change is required to hit this.

Cause

In src/health/service.ts, runHealthCheckpoint does this:

const now = Date.now()
const repair = getRepairHealthEvidence()
const attestedPeers = await countAttestations(now)
const storage = getStorageMetrics()

evaluateHealth in src/health/state.ts rejects the round when storage.updatedAt, the repair completion, or the previous checkpoint is after now (notInFuture). Repair evidence is read before the await, so it stays in the past. Storage is read after the await. refreshStorageMetrics in src/storage/metrics.ts assigns updatedAt: Date.now() only when the directory walk finishes, which can be while attestation is waiting.

refreshHealthSnapshot then latches checks.clockConsistent false and folds it into checkpointFresh, storageFresh, and repairFresh. One ordering glitch is reported as four freshness failures. docs/operations/monitoring.md describes clockConsistent: false as the host clock moving backwards, which is the conclusion an operator will draw.

The health timer is setInterval(health.checkpointIntervalMs) from process start, so its phase never moves. The disk scan is a wall-clock cron (src/disk-usage.cron.ts) every 30 seconds. Those two stay aligned for the whole uptime. Scan duration only has to land inside the attestation wait, bounded by health.peerAttestationTimeoutMs (default 5 seconds, often much shorter), to fail the round.

This is separate from #41. There, a real repair backlog stops checkpoints and the node becomes stale. Here the repair cycle can be successful, inside health.repairMaxAgeMs, with backlog 0.

Proposed solution

Read storage metrics in the same synchronous stretch as now, before the attestation await:

const now = Date.now()
const repair = getRepairHealthEvidence()
const storage = getStorageMetrics()
const attestedPeers = await countAttestations(now)

On one JavaScript turn, updatedAt cannot move between Date.now() and getStorageMetrics(). A scan that finishes during attestation is picked up on the next checkpoint. The default storage window is 120 seconds and the scan period is 30 seconds, so skipping one in-flight sample does not fail storageFresh.

Keep the future-timestamp rejection. A tolerance on notInFuture would also accept a real backward clock step. Do not widen maxCheckpointAgeMs, storageMaxAgeMs, or repairMaxAgeMs.

Add a regression in which attestation waits until a storage refresh has set updatedAt, then assert the checkpoint still completes when that scan is otherwise inside storageMaxAgeMs. Keep the existing evaluateHealth test that refuses a storage measurement from ahead of the attempt, so a real rollback still fails closed.

主要言語
TypeScript
スター
309
フォーク
5
PR マージ指標
30日以内にマージされた PR はありません

環境構築

  • Dockerfile または Docker Compose ファイルあり
  • プルリクエストのテンプレートなし
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Adamant-im/ipfs-node のほかの issue

Adamant-im/ipfs-node の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。