Bug: Migration step 06 fails on incomplete stamp metadata, causing total data loss

オープン
  • 何か問題がありますか？
#5,216 コメント 2 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

@gacevicljubisa がすでに取り組んでいます。

2025年9月22日 から。

評価

この issue はまだ評価されていません。

説明

seen

Summary

Migration step 06 can fail during upgrade to 2.6.0+ when any chunk lacks complete postage stamp metadata, leaving the database in an inconsistent state where all chunks become unreachable despite remaining in storage.

Expected Behavior

  • Migration should handle missing or incomplete stamp metadata gracefully
  • Failed migration should not leave database in broken state
  • Users should be warned about potential data issues before migration runs
  • Rollback mechanism should exist for failed migrations

Current Behavior

Migration fails with recommendation to "nuke" the database when:

  1. Any BatchRadiusItemV1 chunk cannot compute stampHash
  2. chunkstamp.LoadWithBatchID() fails for any chunk
  3. Database left with deleted old indexes but no new indexes created
  4. All data becomes unreachable (both pinned and regular chunks)

Root Cause

Migration step 06 (pkg/storer/migration/step_06.go:112-118) uses atomic approach:

  stamp, err := chunkstamp.LoadWithBatchID(idxStore, "reserve", batchRadiusItemV1.Address, batchRadiusItemV1.BatchID)
  if err != nil {
      return err  // MIGRATION FAILS - database inconsistent
  }
  // Old item already deleted at line 122, new item creation fails

The migration deletes old schema items before ensuring new schema items can be created successfully.

Steps to Reproduce

  1. Run bee node with incomplete stamp metadata (common in older versions)
  2. Upgrade to 2.6.0+
  3. Migration step 06 fails on first chunk with missing stamp data
  4. Observe error: "It's recommended that the nuke cmd is run to reset the node"
  5. Verify chunks exist in storage but cannot be retrieved via any API

Impact

  • Critical: Total data loss requiring database reset
  • Affects: Nodes upgrading from pre-2.6.0 versions
  • User experience: Forced to choose between upgrade and data preservation

Proposed Solutions

Option 1: Graceful Degradation

  • Skip chunks with missing stamp metadata during migration
  • Log warnings for problematic chunks
  • Allow migration to complete with partial data

Option 2: Two-Phase Migration

  • Phase 1: Validate all chunks can be migrated successfully
  • Phase 2: Only proceed with atomic migration if Phase 1 succeeds
  • Provide clear error reporting on validation failures

Option 3: Stamp Reconstruction

  • Attempt to reconstruct missing stamp metadata from available data
  • Use chunk content and batch information to rebuild stamps
  • Fall back to graceful degradation if reconstruction fails

Required Changes

  • Modify pkg/storer/migration/step_06.go migration logic
  • Add pre-migration validation checks
  • Improve error handling and rollback mechanisms
  • Update migration failure messaging with recovery options
  • Add telemetry to track migration success rates

Additional Context

This issue has caused recurring data loss during bee upgrades, particularly affecting nodes with customer data where database reset is not acceptable. The current "nuke database" recommendation is inadequate for production deployments.

Related Issues

  • Pin eviction protection issue (chunks lost due to eviction) #5215
  • Migration should coordinate with pinning system to prioritize preserving pinned data
主要言語
Go
スター
1.5k
フォーク
390
平均マージ
2日 22時間
マージ済み PR（30日）
24

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ethersphere/bee のほかの issue

ethersphere/bee の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。