Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

BLite.Wasm: a 50 MB database cannot be reopened (out of memory reading the WAL); CheckpointAsync returns at once while a background checkpoint runs

Open
  • Is something wrong?
#292 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
csharp, wasm
Domain
backend, databases, performance

Research direction

Start with src/BLite.Core/Storage/StorageEngine.GroupCommit.cs around background checkpoint startup and src/BLite.Core/Storage/StorageEngine.Recovery.cs around the early return, then inspect src/BLite.Wasm/Transactions/IndexedDbWriteAheadLog.cs during WAL reopening. Reproduce with dotnet test tests/BLite.Wasm.BrowserTests --filter "FullyQualifiedName~Large_database". Done means an awaited checkpoint leaves the WAL empty or bounded, a 50 MB database reopens successfully, and the existing size and backend cases preserve their records with bounded time and memory.

Written by the indexing model from the issue text.

Description

bug package/core severity/medium

A database of 50 MB (52,428 documents of 1,000 bytes) filled in one tab, checkpointed and reloaded cannot be reopened: IndexedDbWriteAheadLog.OpenAsync throws OutOfMemoryException while deserialising the WAL. At 20 MB the reopen succeeds but takes 1.5–2.5 s and 545–865 MB of managed heap, with the WebAssembly memory at 1.2–1.4 GB afterwards. The cause is upstream of the browser: past 16 MB of WAL the engine checkpoints in the background, CheckpointAsync returns in 0 ms while that runs, and thousands of records stay in the WAL for the reopen to read as one JSON string.

Found by the browser harness (tests/BLite.Wasm.BrowserTests, https://github.com/EntglDb/BLite/pull/295), which carries the affected rows as known candidate BW6 until this is fixed.

Repro

Rows Large_database_open_time_and_memory IndexedDb/50MB and Opfs/50MB (fail every run); the 20 MB rows pass and record the cost; the 5 MB rows lose their data in half the runs (that is #289).

using var engine = await BLiteWasm.CreateAsync("large", backend);
// 52,428 documents with a 1,000-byte string, InsertBulkAsync in batches of 500
await engine.CheckpointAsync();          // returns in 0 ms
// reload the page
var engine = await BLiteWasm.CreateAsync("large", backend);   // OutOfMemoryException

dotnet test tests/BLite.Wasm.BrowserTests --filter "FullyQualifiedName~Large_database"

Twelve runs per size (two backends, four publish variants, plus two extra runs), Chromium 153, one-thread browser-wasm runtime:

Database CheckpointAsync took Records left in the WAL Documents after reopen Open time Managed heap growth on open WebAssembly memory after open
5 MB, 5,242 docs 95–135 ms (6 of 12 runs) 0 all 165–180 ms +7.5 MB 66–80 MiB
5 MB, 5,242 docs 0 ms (6 of 12 runs) 1,089–1,119 none 310–450 ms +137–198 MB 362–393 MiB
20 MB, 20,971 docs 0 ms (12 of 12) 4,650–5,705 all 1.5–2.5 s +545–865 MB 1,261–1,358 MiB
50 MB, 52,428 docs 0 ms (12 of 12) 12,448–16,111 out of memory – – 1,162–1,468 MiB at failure 
System.OutOfMemoryException: Out of memory
   at System.Text.Json.JsonSerializer.Deserialize[String[]](String, JsonSerializerOptions)
   at BLite.Wasm.Transactions.IndexedDbWriteAheadLog.OpenAsync()
   at BLite.Wasm.BLiteWasm.CreateAsync(String, WasmStorageBackend, Int32, BLiteKvOptions)

Expected: after CheckpointAsync returns, the WAL is empty or nearly so ("guarantee that the page file is fully up-to-date on disk", src/BLite.Core/BLiteEngine.cs:440-442), and a reopen costs the pages, not the WAL. A 50 MB database is a small one.

Actual: the checkpoint returns at once, the reopen reads the whole WAL into memory three times over (one JSON string, one string[] of base64, the decoded records) and runs out.

Mechanism (verified on 0450b97)

  • Past MaxWalSize (16 MB, src/BLite.Core/Storage/StorageEngine.cs:124) a commit starts a background checkpoint: _ = Task.Run(() => CheckpointAsync()) (StorageEngine.GroupCommit.cs:292). While it runs, an explicit CheckpointAsync returns at once (StorageEngine.Recovery.cs:59, if (Interlocked.CompareExchange(ref _checkpointRunning, 1, 0) != 0) return;). Commits that arrive during the background checkpoint stay in the WAL. On the one-thread runtime the background task runs only when the caller yields, which stretches the window.
  • On reopen IndexedDbWriteAheadLog.OpenAsync reads the entire WAL as a single JSON string of base64 records and deserialises it to string[] (src/BLite.Wasm/Transactions/IndexedDbWriteAheadLog.cs:56-59), then parses every record into memory; IndexedDbPageStorage.OpenAsync and OpfsPageStorage.OpenAsync load every page into the managed cache before the engine starts. Each page image in the WAL is 16 KB base64-encoded, so 16,000 records are over 350 MB of strings before decoding.
  • The same CheckpointAsync behaviour exists on desktop; there the WAL is a file read in a stream, so the cost is time rather than memory.

Affected versions: BLite.Wasm since #63 (830e83e, v4.4.0) through v5.2.3; the background checkpoint and the early return since group commit (d72b73a, v3.6.2).

Fix considerations

  • CheckpointAsync waits for a running checkpoint and then checkpoints what arrived since, so its guarantee holds (shared with the reload issue); on desktop as well.
  • Read the browser WAL record by record (a cursor over the object store, or a paged idbWalReadAll) and replay as it goes, instead of one JSON string; store records as ArrayBuffer rather than base64 to halve the size.
  • Do not load every page into memory on open: read pages on demand (ReadPageAsync exists on IPageStorage), or bound the cache. That also bounds the 20 MB cost.
  • The harness records open time and memory per size; the numbers above are the baseline.

Related: #260 (the WAL flush on desktop), #209 / #242 (the checkpoint on dispose), #57 / #63 (BLite.Wasm). #289 comes from the same harness and shares the 0 ms checkpoint; the others are #290, #291 and #293.

Dominant language
C#
Stars
66
Forks
6
Avg merge
1d 2h
Merged PRs (30d)
58

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from EntglDb/BLite

All issues in EntglDb/BLite

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.