[Bug] In-flight disk scan falsely fails health clockConsistent

Đang mở
#49 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức phù hợp với người mới
78/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, typescript

Hướng nghiên cứu

Read src/health/service.ts and src/health/state.ts to follow checkpoint timing and the future timestamp check, then inspect src/storage/metrics.ts and the existing health tests. Reproduce the race by holding attestation until storage metrics refresh, and verify that a scan completed during the wait is deferred to the next checkpoint while a genuinely future timestamp still fails. Done means the regression passes without widening freshness windows.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

APIs bug NodeJS

Summary

runHealthCheckpoint records now before awaiting peer attestation, then reads storage metrics after that await. A disk scan that finishes in between stores updatedAt slightly ahead of now. evaluateHealth treats that as a clock rollback, refuses the checkpoint, and GET /api/node/health stays degraded until the next successful round. Replication, peer attestations, and the host clock can all be healthy. Widening freshness windows, or debouncing a monitor that alerts on state != ready, does not fix it.

Steps to reproduce

  1. Run ipfs-node v0.1.0 with more than one configured node, so health.requiredPeerCount is at least 1 (the default). Leave the health intervals and diskUsageScanPeriod at their defaults: a checkpoint every 60 seconds, a disk scan every 30 seconds.
  2. Let the storage scan run long enough that it sometimes completes while countAttestations is in flight. A blockstore directory walk of several seconds is enough when the process timer phase sits near scan completion.
  3. Poll GET /api/node/health across a few checkpoint intervals.

A deterministic test can hold attestation until refreshStorageMetrics has set updatedAt, then release it.

Expected behavior

A storage measurement taken during the checkpoint attempt is either included with an evaluation time that is not earlier than updatedAt, or left for the next round. checks.clockConsistent stays true unless the clock actually moved behind a timestamp this process already recorded. With a fresh scan, a successful repair cycle, and the required peer attestations, state stays ready.

Actual behavior

The checkpoint fails closed. Until the next successful round (up to health.checkpointIntervalMs):

  • state is degraded while checkpoint.ageMs is still below checkpoint.maxAgeMs
  • storage.measuredAt is a few milliseconds after evaluatedAt, while storage.measurementAgeMs at read time is still well inside health.storageMaxAgeMs
  • checks.clockConsistent is false. Read-time refresh will not set it back to true, so later polls keep the failure even after wall time has moved past storage.measuredAt
  • checks.checkpointFresh, checks.storageFresh, and checks.repairFresh are false with it, because refresh requires clockConsistent for all three
  • replication.repairRequired stays true. That flag means repair is part of the readiness contract (replication.enabled and replication.repairEnabled), not that the last cycle failed. A successful cycle with backlog 0 still reports repairRequired: true
  • checks.helia, checks.startupReconciliation, checks.storageReserve, and checks.peerAttestations stay true
  • timestamp minus evaluatedAt grows through the interval. That gap is the age of the cached snapshot, including while state is ready

The next checkpoint that does not overlap a scan completion returns ready. Each overlap therefore produces about a minute of degraded, and the process timer keeps the same phase for its whole lifetime, so the flap repeats until the phase no longer meets scan completion. A restart only draws a new phase.

Environment

ipfs-node v0.1.0, current health checkpoint code. Defaults: health.checkpointIntervalMs 60000, health.maxCheckpointAgeMs 180000, health.storageMaxAgeMs 120000, health.repairMaxAgeMs 3600000, diskUsageScanPeriod */30 * * * * *, repair enabled, one required peer. No configuration change is required to hit this.

Cause

In src/health/service.ts, runHealthCheckpoint does this:

const now = Date.now()
const repair = getRepairHealthEvidence()
const attestedPeers = await countAttestations(now)
const storage = getStorageMetrics()

evaluateHealth in src/health/state.ts rejects the round when storage.updatedAt, the repair completion, or the previous checkpoint is after now (notInFuture). Repair evidence is read before the await, so it stays in the past. Storage is read after the await. refreshStorageMetrics in src/storage/metrics.ts assigns updatedAt: Date.now() only when the directory walk finishes, which can be while attestation is waiting.

refreshHealthSnapshot then latches checks.clockConsistent false and folds it into checkpointFresh, storageFresh, and repairFresh. One ordering glitch is reported as four freshness failures. docs/operations/monitoring.md describes clockConsistent: false as the host clock moving backwards, which is the conclusion an operator will draw.

The health timer is setInterval(health.checkpointIntervalMs) from process start, so its phase never moves. The disk scan is a wall-clock cron (src/disk-usage.cron.ts) every 30 seconds. Those two stay aligned for the whole uptime. Scan duration only has to land inside the attestation wait, bounded by health.peerAttestationTimeoutMs (default 5 seconds, often much shorter), to fail the round.

This is separate from #41. There, a real repair backlog stops checkpoints and the node becomes stale. Here the repair cycle can be successful, inside health.repairMaxAgeMs, with backlog 0.

Proposed solution

Read storage metrics in the same synchronous stretch as now, before the attestation await:

const now = Date.now()
const repair = getRepairHealthEvidence()
const storage = getStorageMetrics()
const attestedPeers = await countAttestations(now)

On one JavaScript turn, updatedAt cannot move between Date.now() and getStorageMetrics(). A scan that finishes during attestation is picked up on the next checkpoint. The default storage window is 120 seconds and the scan period is 30 seconds, so skipping one in-flight sample does not fail storageFresh.

Keep the future-timestamp rejection. A tolerance on notInFuture would also accept a real backward clock step. Do not widen maxCheckpointAgeMs, storageMaxAgeMs, or repairMaxAgeMs.

Add a regression in which attestation waits until a storage refresh has set updatedAt, then assert the checkpoint still completes when that scan is otherwise inside storageMaxAgeMs. Keep the existing evaluateHealth test that refuses a storage measurement from ahead of the attempt, so a real rollback still fails closed.

Ngôn ngữ chính
TypeScript
Star
309
Fork
5
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

  • Có Dockerfile hoặc tệp Docker Compose
  • Không có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Adamant-im/ipfs-node

Tất cả issue của Adamant-im/ipfs-node

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.