Make production purge and offset recovery testable with SimStorage

Open
#4,239 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
rust

Research direction

Start with core/simulator/src/storage/tests.rs and the cluster restart path in core/simulator/src/lib.rs, then trace DurableStorage, SimStorage, and the existing storage purge test. Run the focused simulator storage tests first. Done means production cleanup and recovery are shared with a focused harness, with passing replicated and persisted offset durability checks across simulated power loss.

Written by the indexing model from the issue text.

Description

Following hubcio's suggestion in #4128, make production purge cleanup, completion and consumer offset recovery usable with SimStorage through DurableStorage. This would let us test what survives power loss before relying on a fix for #4128.

SimStorage already models directory sync failures and Crash::PowerLoss, including restoring file entries whose deletion was not synced. The missing part is connecting production code to that storage. The existing storage purge test implements its own cleanup sequence and propagates sync errors, while production logs those errors and continues. The cluster restart path retains consumer offsets in memory. Neither path currently exercises the recovery sequence needed for this bug.

The implementation should

  • Share the production cleanup and completion control flow with a focused storage harness, including the decisions made after errors. Sharing only the filesystem calls would still let the test behave differently from production.
  • Route the relevant directory scans, offset file deletion, directory syncs and persistence of purge.gen through DurableStorage.
  • Preserve separate directories for individual consumer offsets, consumer group offsets and the generation marker. Syncing the marker's directory must not make deletions in the offset directories durable.
  • Reload purge.gen and both kinds of consumer offsets from the simulated filesystem through shared production recovery code. Checks of Next must use that recovered state.
  • Add passing controls for successful purge and recovery under both replicated and persisted consumer offset durability. Check individual consumers and consumer groups, and verify that Next returns the complete fresh history. Make that history durable before any simulated power loss so these checks isolate offset recovery from message durability.

Keep this as a preparatory refactor that preserves production behavior. A focused harness using the existing storage simulator is enough to start. Converting the whole cluster simulator is not required.

The regression for #4128 and #4130 can then fail each offset directory sync separately after successful deletion, allow subsequent operations and simulate power loss. That regression should check that the completion marker cannot suppress required cleanup, stale offsets disappear after cleanup, and retries preserve acknowledged fresh writes. #4130 already reproduces the production sync failure but does not cover that full recovery sequence.

Dominant language
Rust
Stars
4.9k
Forks
442
Avg merge
2d 2h
Merged PRs (30d)
160

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iggy

All issues in apache/iggy

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.