Cache spool: active.jsonl has no cross-process lock, so a failed append's rollback can discard a foreign record

Open
  • Is something wrong?
#2,774 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript
Domain
backend, databases

Research direction

Read src/core/cache/spool.js and the CacheSpool.append contract in src/core/cache/types.d.ts, then trace discardTail, rotation, and drain operations. Decide which operations need a cross-process advisory lock, whether locking is per table or per cache, and how callers should handle lock acquisition failure; the issue is complete when those design choices are documented as a request for implementation.

Written by the indexing model from the issue text.

Description

Split out of #939 (finding 2), as Phil asked there: "file the lock as its own request LLP (or a fresh issue) rather than reopening this one."

Problem

src/core/cache/spool.js discardTail rolls back a failed append by checking that the tail bytes are its own, then truncating. On current master that narrows the cross-process hazard on a table's active.jsonl but does not close it:

  • Reading the tail (handle.stat / handle.read) and truncating (handle.truncate) are separate syscalls. A foreign record appended between them sits inside the truncated range and is discarded, though its writer was told it landed.
  • A torn prefix landing before a foreign line concatenates the two into one malformed line, which the flush reader drops.

Both windows need a cross-process advisory lock (e.g. flock) over active.jsonl. src/core/cache/ has no such lock today; the in-process withWriteLock only serializes one process's writers. The residual windows are documented at discardTail and in the CacheSpool.append contract in src/core/cache/types.d.ts.

What a request LLP has to settle before code

  • Which files and operations take the lock (append only, or rotation and drain too).
  • Per table or per cache.
  • What a caller does when the lock cannot be taken.

Severity: low, design question. Not a patch.

🤖 Generated with Claude Code

Dominant language
JavaScript
Stars
11
Forks
0
Avg merge
2h 55m
Merged PRs (30d)
294

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hyparam/hypaware

All issues in hyparam/hypaware

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.