Corrupt cleanup journal blocks a partition's compaction and snapshot expiry, not only reclamation

Open
  • Is something wrong?
#2,022 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript
Domain
backend, databases

Research direction

Read src/core/cache/maintenance.js at lines 442 and 660, then the cited design decision in llp/0417-session-purge-across-storage.spec.md lines 233–235 and the referenced LLP 0220 partition-walk behavior. First decide through a new or superseding LLP whether maintenance should continue after an unreadable journal or expose a health signal. Done means implementing that decision and adding a corrupt-journal regression test.

Written by the indexing model from the issue text.

Description

Source: #2017 at 6bd3752d4a7e0f3fa70881e5f5e6a8c42aff5854
Location: src/core/cache/maintenance.js:442 (and :660)
Severity: medium

Evidence

maintainGeneration calls readCacheCleanup un-caught at its top (maintenance.js:442), ahead of the compaction and snapshot-expiry work, so a journal that is present but unparseable fails that partition's whole maintenance tick for as long as it stays corrupt; ongoing appends then accumulate snapshots and superseded files in that partition until a later purge of the same partition rebuilds the journal. The blast radius is one partition: the catch at maintenance.js:304 (@ref LLP 0220#walk-survives-a-partition) marks the partition failed and lets the walk continue, and walkForRetired independently catches its own read (maintenance.js:2672).

Why deferral is safe

At this head the behavior is documented design, not a spec-code contradiction: the sentence this PR corrected (llp/0417-session-purge-across-storage.spec.md:233-235) now states that an invalid journal blocks that partition's maintenance, compaction and snapshot expiry included. Changing it, catching the read and proceeding with compaction while refusing only reclamation, is a revisit of a recorded fail-closed decision and needs an LLP, not a code patch. The trigger is rare (journals are written by atomicWriteJson with fsync, tmp plus rename, so a torn journal needs a hand edit or unusual filesystem behavior), the failure is isolated per partition and surfaced as a failed partition report, and a purge of that partition self-heals it.

Acceptance condition

A design decision (new LLP extending or superseding the relevant part of LLP 0417) on whether maintainGeneration should survive an unreadable journal, blocking only reclamation, or whether the blocked-partition state should instead be surfaced as a health signal; then implement per that decision with a corrupt-journal regression test.

Review preference; backlog only. A human may explicitly delegate it with neutral:fix.

Dominant language
JavaScript
Stars
11
Forks
0
Avg merge
2h 49m
Merged PRs (30d)
302

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hyparam/hypaware

All issues in hyparam/hypaware

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.