[Bug] In-flight disk scan falsely fails health clockConsistent
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- node.js, typescript
- Domain
- backend, observability, testing-qa
Research direction
Read src/health/service.ts and src/health/state.ts to follow checkpoint timing and the future timestamp check, then inspect src/storage/metrics.ts and the existing health tests. Reproduce the race by holding attestation until storage metrics refresh, and verify that a scan completed during the wait is deferred to the next checkpoint while a genuinely future timestamp still fails. Done means the regression passes without widening freshness windows.
Written by the indexing model from the issue text.
Description
Summary
runHealthCheckpoint records now before awaiting peer attestation, then reads storage metrics after that await. A disk scan that finishes in between stores updatedAt slightly ahead of now. evaluateHealth treats that as a clock rollback, refuses the checkpoint, and GET /api/node/health stays degraded until the next successful round. Replication, peer attestations, and the host clock can all be healthy. Widening freshness windows, or debouncing a monitor that alerts on state != ready, does not fix it.
Steps to reproduce
- Run ipfs-node v0.1.0 with more than one configured node, so
health.requiredPeerCountis at least 1 (the default). Leave the health intervals anddiskUsageScanPeriodat their defaults: a checkpoint every 60 seconds, a disk scan every 30 seconds. - Let the storage scan run long enough that it sometimes completes while
countAttestationsis in flight. A blockstore directory walk of several seconds is enough when the process timer phase sits near scan completion. - Poll
GET /api/node/healthacross a few checkpoint intervals.
A deterministic test can hold attestation until refreshStorageMetrics has set updatedAt, then release it.
Expected behavior
A storage measurement taken during the checkpoint attempt is either included with an evaluation time that is not earlier than updatedAt, or left for the next round. checks.clockConsistent stays true unless the clock actually moved behind a timestamp this process already recorded. With a fresh scan, a successful repair cycle, and the required peer attestations, state stays ready.
Actual behavior
The checkpoint fails closed. Until the next successful round (up to health.checkpointIntervalMs):
stateisdegradedwhilecheckpoint.ageMsis still belowcheckpoint.maxAgeMsstorage.measuredAtis a few milliseconds afterevaluatedAt, whilestorage.measurementAgeMsat read time is still well insidehealth.storageMaxAgeMschecks.clockConsistentis false. Read-time refresh will not set it back to true, so later polls keep the failure even after wall time has moved paststorage.measuredAtchecks.checkpointFresh,checks.storageFresh, andchecks.repairFreshare false with it, because refresh requiresclockConsistentfor all threereplication.repairRequiredstays true. That flag means repair is part of the readiness contract (replication.enabledandreplication.repairEnabled), not that the last cycle failed. A successful cycle with backlog 0 still reportsrepairRequired: truechecks.helia,checks.startupReconciliation,checks.storageReserve, andchecks.peerAttestationsstay truetimestampminusevaluatedAtgrows through the interval. That gap is the age of the cached snapshot, including whilestateisready
The next checkpoint that does not overlap a scan completion returns ready. Each overlap therefore produces about a minute of degraded, and the process timer keeps the same phase for its whole lifetime, so the flap repeats until the phase no longer meets scan completion. A restart only draws a new phase.
Environment
ipfs-node v0.1.0, current health checkpoint code. Defaults: health.checkpointIntervalMs 60000, health.maxCheckpointAgeMs 180000, health.storageMaxAgeMs 120000, health.repairMaxAgeMs 3600000, diskUsageScanPeriod */30 * * * * *, repair enabled, one required peer. No configuration change is required to hit this.
Cause
In src/health/service.ts, runHealthCheckpoint does this:
const now = Date.now()
const repair = getRepairHealthEvidence()
const attestedPeers = await countAttestations(now)
const storage = getStorageMetrics()
evaluateHealth in src/health/state.ts rejects the round when storage.updatedAt, the repair completion, or the previous checkpoint is after now (notInFuture). Repair evidence is read before the await, so it stays in the past. Storage is read after the await. refreshStorageMetrics in src/storage/metrics.ts assigns updatedAt: Date.now() only when the directory walk finishes, which can be while attestation is waiting.
refreshHealthSnapshot then latches checks.clockConsistent false and folds it into checkpointFresh, storageFresh, and repairFresh. One ordering glitch is reported as four freshness failures. docs/operations/monitoring.md describes clockConsistent: false as the host clock moving backwards, which is the conclusion an operator will draw.
The health timer is setInterval(health.checkpointIntervalMs) from process start, so its phase never moves. The disk scan is a wall-clock cron (src/disk-usage.cron.ts) every 30 seconds. Those two stay aligned for the whole uptime. Scan duration only has to land inside the attestation wait, bounded by health.peerAttestationTimeoutMs (default 5 seconds, often much shorter), to fail the round.
This is separate from #41. There, a real repair backlog stops checkpoints and the node becomes stale. Here the repair cycle can be successful, inside health.repairMaxAgeMs, with backlog 0.
Proposed solution
Read storage metrics in the same synchronous stretch as now, before the attestation await:
const now = Date.now()
const repair = getRepairHealthEvidence()
const storage = getStorageMetrics()
const attestedPeers = await countAttestations(now)
On one JavaScript turn, updatedAt cannot move between Date.now() and getStorageMetrics(). A scan that finishes during attestation is picked up on the next checkpoint. The default storage window is 120 seconds and the scan period is 30 seconds, so skipping one in-flight sample does not fail storageFresh.
Keep the future-timestamp rejection. A tolerance on notInFuture would also accept a real backward clock step. Do not widen maxCheckpointAgeMs, storageMaxAgeMs, or repairMaxAgeMs.
Add a regression in which attestation waits until a storage refresh has set updatedAt, then assert the checkpoint still completes when that scan is otherwise inside storageMaxAgeMs. Keep the existing evaluateHealth test that refuses a storage measurement from ahead of the attempt, so a real rollback still fails closed.
- Dominant language
- TypeScript
- Stars
- 309
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Adamant-im/ipfs-node
-
Decentralization IPFS Privacy Research Security
Difficulty 5/5 Over a week Newbie friendliness 35/100
Adamant-im/ipfs-node#31 ·
-
enhancement Infrastructure IPFS
Difficulty 5/5 Over a week Newbie friendliness 38/100
Adamant-im/ipfs-node#30 ·
-
enhancement Infrastructure IPFS
Difficulty 5/5 Over a week Newbie friendliness 30/100
Adamant-im/ipfs-node#29 ·
-
Composite task Decentralization Infrastructure IPFS
Difficulty 5/5 Over a week Newbie friendliness 25/100
Adamant-im/ipfs-node#28 ·
-
Composite task Decentralization Encryption Governance IPFS
Difficulty 5/5 Over a week Newbie friendliness 25/100
Adamant-im/ipfs-node#27 · 1 comment ·
All issues in Adamant-im/ipfs-node
Similar issues
-
Bug Tests
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
JSPrismarine/JSPrismarine#3384 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
zingolabs/zingo-mobile#1951 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Automattic/data-liberation-agent#2175 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
xdelmo/Portfolio-minimal#613 ·
Maintainers usually reply within 1 day
-
backend documentation low-priority Stellar Wave
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Savitura/Savitools#611 · 1 comment ·
Maintainers usually reply within 6 days