[Bug] In-flight disk scan falsely fails health clockConsistent

Open
#49 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
node.js, typescript

Research direction

Read src/health/service.ts and src/health/state.ts to follow checkpoint timing and the future timestamp check, then inspect src/storage/metrics.ts and the existing health tests. Reproduce the race by holding attestation until storage metrics refresh, and verify that a scan completed during the wait is deferred to the next checkpoint while a genuinely future timestamp still fails. Done means the regression passes without widening freshness windows.

Written by the indexing model from the issue text.

Description

APIs bug NodeJS

Summary

runHealthCheckpoint records now before awaiting peer attestation, then reads storage metrics after that await. A disk scan that finishes in between stores updatedAt slightly ahead of now. evaluateHealth treats that as a clock rollback, refuses the checkpoint, and GET /api/node/health stays degraded until the next successful round. Replication, peer attestations, and the host clock can all be healthy. Widening freshness windows, or debouncing a monitor that alerts on state != ready, does not fix it.

Steps to reproduce

  1. Run ipfs-node v0.1.0 with more than one configured node, so health.requiredPeerCount is at least 1 (the default). Leave the health intervals and diskUsageScanPeriod at their defaults: a checkpoint every 60 seconds, a disk scan every 30 seconds.
  2. Let the storage scan run long enough that it sometimes completes while countAttestations is in flight. A blockstore directory walk of several seconds is enough when the process timer phase sits near scan completion.
  3. Poll GET /api/node/health across a few checkpoint intervals.

A deterministic test can hold attestation until refreshStorageMetrics has set updatedAt, then release it.

Expected behavior

A storage measurement taken during the checkpoint attempt is either included with an evaluation time that is not earlier than updatedAt, or left for the next round. checks.clockConsistent stays true unless the clock actually moved behind a timestamp this process already recorded. With a fresh scan, a successful repair cycle, and the required peer attestations, state stays ready.

Actual behavior

The checkpoint fails closed. Until the next successful round (up to health.checkpointIntervalMs):

  • state is degraded while checkpoint.ageMs is still below checkpoint.maxAgeMs
  • storage.measuredAt is a few milliseconds after evaluatedAt, while storage.measurementAgeMs at read time is still well inside health.storageMaxAgeMs
  • checks.clockConsistent is false. Read-time refresh will not set it back to true, so later polls keep the failure even after wall time has moved past storage.measuredAt
  • checks.checkpointFresh, checks.storageFresh, and checks.repairFresh are false with it, because refresh requires clockConsistent for all three
  • replication.repairRequired stays true. That flag means repair is part of the readiness contract (replication.enabled and replication.repairEnabled), not that the last cycle failed. A successful cycle with backlog 0 still reports repairRequired: true
  • checks.helia, checks.startupReconciliation, checks.storageReserve, and checks.peerAttestations stay true
  • timestamp minus evaluatedAt grows through the interval. That gap is the age of the cached snapshot, including while state is ready

The next checkpoint that does not overlap a scan completion returns ready. Each overlap therefore produces about a minute of degraded, and the process timer keeps the same phase for its whole lifetime, so the flap repeats until the phase no longer meets scan completion. A restart only draws a new phase.

Environment

ipfs-node v0.1.0, current health checkpoint code. Defaults: health.checkpointIntervalMs 60000, health.maxCheckpointAgeMs 180000, health.storageMaxAgeMs 120000, health.repairMaxAgeMs 3600000, diskUsageScanPeriod */30 * * * * *, repair enabled, one required peer. No configuration change is required to hit this.

Cause

In src/health/service.ts, runHealthCheckpoint does this:

const now = Date.now()
const repair = getRepairHealthEvidence()
const attestedPeers = await countAttestations(now)
const storage = getStorageMetrics()

evaluateHealth in src/health/state.ts rejects the round when storage.updatedAt, the repair completion, or the previous checkpoint is after now (notInFuture). Repair evidence is read before the await, so it stays in the past. Storage is read after the await. refreshStorageMetrics in src/storage/metrics.ts assigns updatedAt: Date.now() only when the directory walk finishes, which can be while attestation is waiting.

refreshHealthSnapshot then latches checks.clockConsistent false and folds it into checkpointFresh, storageFresh, and repairFresh. One ordering glitch is reported as four freshness failures. docs/operations/monitoring.md describes clockConsistent: false as the host clock moving backwards, which is the conclusion an operator will draw.

The health timer is setInterval(health.checkpointIntervalMs) from process start, so its phase never moves. The disk scan is a wall-clock cron (src/disk-usage.cron.ts) every 30 seconds. Those two stay aligned for the whole uptime. Scan duration only has to land inside the attestation wait, bounded by health.peerAttestationTimeoutMs (default 5 seconds, often much shorter), to fail the round.

This is separate from #41. There, a real repair backlog stops checkpoints and the node becomes stale. Here the repair cycle can be successful, inside health.repairMaxAgeMs, with backlog 0.

Proposed solution

Read storage metrics in the same synchronous stretch as now, before the attestation await:

const now = Date.now()
const repair = getRepairHealthEvidence()
const storage = getStorageMetrics()
const attestedPeers = await countAttestations(now)

On one JavaScript turn, updatedAt cannot move between Date.now() and getStorageMetrics(). A scan that finishes during attestation is picked up on the next checkpoint. The default storage window is 120 seconds and the scan period is 30 seconds, so skipping one in-flight sample does not fail storageFresh.

Keep the future-timestamp rejection. A tolerance on notInFuture would also accept a real backward clock step. Do not widen maxCheckpointAgeMs, storageMaxAgeMs, or repairMaxAgeMs.

Add a regression in which attestation waits until a storage refresh has set updatedAt, then assert the checkpoint still completes when that scan is otherwise inside storageMaxAgeMs. Keep the existing evaluateHealth test that refuses a storage measurement from ahead of the attempt, so a real rollback still fails closed.

Dominant language
TypeScript
Stars
309
Forks
5
PR merge metrics
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Adamant-im/ipfs-node

All issues in Adamant-im/ipfs-node

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.