Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Bug: Migration step 06 fails on incomplete stamp metadata, causing total data loss

Open
  • Is something wrong?
#5,216 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 2 days

@gacevicljubisa is already working on this.

Since Sep 22, 2025.

Assessment

This issue has not been assessed yet.

Description

Summary

Migration step 06 can fail during upgrade to 2.6.0+ when any chunk lacks complete postage stamp metadata, leaving the database in an inconsistent state where all chunks become unreachable despite remaining in storage.

Expected Behavior

  • Migration should handle missing or incomplete stamp metadata gracefully
  • Failed migration should not leave database in broken state
  • Users should be warned about potential data issues before migration runs
  • Rollback mechanism should exist for failed migrations

Current Behavior

Migration fails with recommendation to "nuke" the database when:

  1. Any BatchRadiusItemV1 chunk cannot compute stampHash
  2. chunkstamp.LoadWithBatchID() fails for any chunk
  3. Database left with deleted old indexes but no new indexes created
  4. All data becomes unreachable (both pinned and regular chunks)

Root Cause

Migration step 06 (pkg/storer/migration/step_06.go:112-118) uses atomic approach:

  stamp, err := chunkstamp.LoadWithBatchID(idxStore, "reserve", batchRadiusItemV1.Address, batchRadiusItemV1.BatchID)
  if err != nil {
      return err  // MIGRATION FAILS - database inconsistent
  }
  // Old item already deleted at line 122, new item creation fails

The migration deletes old schema items before ensuring new schema items can be created successfully.

Steps to Reproduce

  1. Run bee node with incomplete stamp metadata (common in older versions)
  2. Upgrade to 2.6.0+
  3. Migration step 06 fails on first chunk with missing stamp data
  4. Observe error: "It's recommended that the nuke cmd is run to reset the node"
  5. Verify chunks exist in storage but cannot be retrieved via any API

Impact

  • Critical: Total data loss requiring database reset
  • Affects: Nodes upgrading from pre-2.6.0 versions
  • User experience: Forced to choose between upgrade and data preservation

Proposed Solutions

Option 1: Graceful Degradation

  • Skip chunks with missing stamp metadata during migration
  • Log warnings for problematic chunks
  • Allow migration to complete with partial data

Option 2: Two-Phase Migration

  • Phase 1: Validate all chunks can be migrated successfully
  • Phase 2: Only proceed with atomic migration if Phase 1 succeeds
  • Provide clear error reporting on validation failures

Option 3: Stamp Reconstruction

  • Attempt to reconstruct missing stamp metadata from available data
  • Use chunk content and batch information to rebuild stamps
  • Fall back to graceful degradation if reconstruction fails

Required Changes

  • Modify pkg/storer/migration/step_06.go migration logic
  • Add pre-migration validation checks
  • Improve error handling and rollback mechanisms
  • Update migration failure messaging with recovery options
  • Add telemetry to track migration success rates

Additional Context

This issue has caused recurring data loss during bee upgrades, particularly affecting nodes with customer data where database reset is not acceptable. The current "nuke database" recommendation is inadequate for production deployments.

Related Issues

  • Pin eviction protection issue (chunks lost due to eviction) #5215
  • Migration should coordinate with pinning system to prioritize preserving pinned data
Dominant language
Go
Stars
1.5k
Forks
390
Avg merge
2d 11h
Merged PRs (30d)
23

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ethersphere/bee

All issues in ethersphere/bee

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.