Make production purge and offset recovery testable with SimStorage
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 38/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend, databases, testing-qa
Research direction
Start with core/simulator/src/storage/tests.rs and the cluster restart path in core/simulator/src/lib.rs, then trace DurableStorage, SimStorage, and the existing storage purge test. Run the focused simulator storage tests first. Done means production cleanup and recovery are shared with a focused harness, with passing replicated and persisted offset durability checks across simulated power loss.
Written by the indexing model from the issue text.
Description
Following hubcio's suggestion in #4128, make production purge cleanup, completion and consumer offset recovery usable with SimStorage through DurableStorage. This would let us test what survives power loss before relying on a fix for #4128.
SimStorage already models directory sync failures and Crash::PowerLoss, including restoring file entries whose deletion was not synced. The missing part is connecting production code to that storage. The existing storage purge test implements its own cleanup sequence and propagates sync errors, while production logs those errors and continues. The cluster restart path retains consumer offsets in memory. Neither path currently exercises the recovery sequence needed for this bug.
The implementation should
- Share the production cleanup and completion control flow with a focused storage harness, including the decisions made after errors. Sharing only the filesystem calls would still let the test behave differently from production.
- Route the relevant directory scans, offset file deletion, directory syncs and persistence of
purge.genthroughDurableStorage. - Preserve separate directories for individual consumer offsets, consumer group offsets and the generation marker. Syncing the marker's directory must not make deletions in the offset directories durable.
- Reload
purge.genand both kinds of consumer offsets from the simulated filesystem through shared production recovery code. Checks ofNextmust use that recovered state. - Add passing controls for successful purge and recovery under both
replicatedandpersistedconsumer offset durability. Check individual consumers and consumer groups, and verify thatNextreturns the complete fresh history. Make that history durable before any simulated power loss so these checks isolate offset recovery from message durability.
Keep this as a preparatory refactor that preserves production behavior. A focused harness using the existing storage simulator is enough to start. Converting the whole cluster simulator is not required.
The regression for #4128 and #4130 can then fail each offset directory sync separately after successful deletion, allow subsequent operations and simulate power loss. That regression should check that the completion marker cannot suppress required cleanup, stale offsets disappear after cleanup, and retries preserve acknowledged fresh writes. #4130 already reproduces the production sync failure but does not cover that full recovery sequence.
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 442
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 160
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iggy
-
server test
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
connectors: runtime discards the sink consume FFI status, so failed batches are counted as processed Openbug connectors rust
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
bug python sdk
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
cluster server
-
connectors
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100