Multi-process WAL: a permanently unreadable record inside the published range stalls replay, checkpoints and truncation
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 8/100
Research direction
The stall lives in the multi-process WAL catch-up path: the replay over [bookmark, SHM WalEndOffset), the _walIndex and checkpoint code, and the AdvanceWalEndOffset and RecoverAsync truncation on open (#200). Read those alongside the trace warning and MetricsSnapshot.CheckpointsFailedTotal. Done would need a decision on whether to fix #200 or add a recovery path past a bad record, plus tests reproducing a permanently unreadable record.
Written by the indexing model from the issue text.
Description
Found during the review of #202.
A process catching up with the shared WAL replays [bookmark, SHM WalEndOffset). Since #202, a record it can't read (torn, corrupt, or past the end of the file) ends the catch-up as incomplete, and the bookmark stays put. That avoids skipping or truncating later records (#198). But if the record stays unreadable:
- every catch-up (BeginTransaction, group commit, checkpoint) fails for every process whose bookmark is before it;
- those processes' checkpoints write nothing, not even their own pages, and they never truncate the WAL;
- their
_walIndexand the WAL grow without bound, and their reads may miss other processes' commits.
The stall only ends if the record becomes readable again, or if a process whose bookmark is already past the record (e.g. the one that wrote it) checkpoints and truncates the WAL.
Causes:
- disk corruption inside the published range;
- a record written at a stale offset by a path that bypasses the locked append sequence (#188; these paths now throw in multi-process mode);
- #200: after the unlocked
RecoverAsynctruncation on open, the SHMWalEndOffsetstays past the file's end, becauseAdvanceWalEndOffsetonly moves forward. Every replay is then incomplete until the WAL grows past the old end. This is a stall, not a loss.
Since #202 the stall is visible. A trace warning gives the WAL path and the offset: the first time, then every power of two in a row, and from 16 in a row it says "stalled". Failed checkpoints are counted in MetricsSnapshot.CheckpointsFailedTotal. Nothing recovers from it automatically.
Possible directions:
- Fix #200: truncate under the SHM writer lock and reset the end offset, or skip the truncation on open while other processes are attached.
- After a stall threshold, let a process that holds the writer lock and has checkpointed everything it can read trigger a recovery: replay the WAL up to the bad record, checkpoint, truncate, start a new generation. This deliberately gives up the records past the bad one, so it needs a decision on whether that's acceptable or whether to stop with an error.
- Expose the stall as a metric (e.g. a consecutive-failed-catch-up gauge), not only as a trace warning.
- Dominant language
- C#
- Stars
- 66
- Forks
- 6
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 58
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from EntglDb/BLite
-
GdprMode.Strict warns that deleted documents are not zeroed, but every delete zero-fills themPossibly taken @CaffeinatedCoder claimed this 3 days ago. Openbug gdpr package/core severity/low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
EntglDb/BLite#299 · 1 comment ·
Maintainers usually reply within 2 days
-
bug package/core severity/medium triage
Difficulty 4/5 3-5 days Newbie friendliness 54/100
Maintainers usually reply within 2 days
-
bug package/core severity/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
EntglDb/BLite#322 · 1 comment ·
Maintainers usually reply within 2 days
-
enhancement package/core
Difficulty 4/5 3-5 days Newbie friendliness 48/100
EntglDb/BLite#320 · 1 comment ·
Maintainers usually reply within 2 days
-
BLiteSession.Rollback() does not await the rollback: it returns while the transaction is still active, and failures go unobservedPossibly taken @CaffeinatedCoder claimed this 3 days ago. Openbug package/core severity/medium
Difficulty 3/5 1-2 days Newbie friendliness 78/100
EntglDb/BLite#307 · 1 comment ·
Maintainers usually reply within 2 days
Similar issues
-
/api/oauth/introspect emits clientId and tokenType instead of RFC 7662's client_id and token_typeOpenbug P2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
nightscout/nocturne#2317 ·
Maintainers usually reply within 1 day
-
[BUG]: Position intent set on a simple order is dropped when it becomes a bracket, OCO or OTO orderPossibly taken @RaphaelFakhri claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
alpacahq/alpaca-trade-api-csharp#903 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
joinrpg/joinrpg-net#5752 ·
Maintainers usually reply within 1 day
-
bug ux
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
CodeForFire/lagebuch#714 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
dimitar-radenkov/Pointframe#294 ·
Maintainers usually reply within 1 day