Cache reconcile can skip retirement when a dataset's cachePartitioning chain changes, leaving duplicate rows

Open
  • Is something wrong?
#2,332 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript
Domain
backend, databases

Research direction

Start with src/core/cache/reconcile.js, especially partitionsOutsideScope, and read the linked PR #2330 discussion to understand the existing guard. Reproduce the chain-change scenario using appendRows and flushAll, as described in the issue. Done means reconciliation can no longer leave duplicate live rows when a partition’s label was written under a different cachePartitioning chain.

Written by the indexing model from the issue text.

Description

partitionsOutsideScope in src/core/cache/reconcile.js decides a partition
provably holds none of a scope's rows by reading its source=<value> segment
and comparing it to the value the scope resolves to. The comparison is only
sound while the resolver that labelled the partition agrees with the one the
scope uses now.

Nothing on disk records which resolver wrote a label. The guard added in
PR #2330 checks declaration.source.columns[0] === 'client_name' against the
declaration in force at reconcile time, but the label was written by
whatever chain was in force at flush time. When a dataset's
cachePartitioning chain changes, old partitions keep labels the current
resolver never produces, and their scope-matching rows are skipped instead of
retired: two live rows for one identity, and silently wrong query results.

Reproduction

Driven entirely through production writers (appendRows + flushAll), never
appendRowsToPartition:

  1. Flush with source.columns = ['provider', 'client_name']. A row
    {client_name: 'hermes', session_id: 'hermes-s', part_id: 'p1'} with
    provider: 'openai' lands in source=openai.
  2. Corrupt that partition's cursor.json (the #2282 condition).
  3. Change the chain to the shipped ['client_name', 'conversation_source', 'provider'] (or remove the declaration), then reconcile the same scope.
reconcile returned 1 (did NOT refuse)
dirs on disk: ["_hypaware_spool","source=hermes","source=openai"]

ROWS AFTER CURSOR REPAIR:
    source=hermes -> new   (part_id=p1, session_id=hermes-s, client_name=hermes)
    source=openai -> stale (part_id=p1, session_id=hermes-s, client_name=hermes)

==> 2 live rows for the SAME identity (part_id=p1, session_id=hermes-s)

Controls: with no drift, both the shipped chain and the provider-first chain
correctly refuse.

Why it is not reachable today

ai-gateway/src/dataset.js:281 is the only cachePartitioning declaration in
the tree and has always led with client_name, so no partition on disk can
carry a disagreeing label. cachePartitioning is public plugin surface, so
this is a trap for a future declaration rather than a live defect, which is
why PR #2330 documents the precondition rather than closing it.

The trap worth naming

The reconcile-time guard tells a plugin author that a chain leading with
client_name is the supported shape. Correcting a chain to lead with
client_name is therefore the natural remediation, and it is exactly the
change that opens this gap on the rows already written under the old chain.

Why there is no small fix
  • Recording the resolved source in cursor.json does not work: this path
    skips a partition precisely because its cursor is unreadable, so anything
    stored there is unavailable when it is needed.
  • Reading the partition's rows to check is what the refusal exists to avoid
    (the live generation is unknown).
  • No predicate over the label alone can distinguish a label written by an
    agreeing resolver from one written by a disagreeing one.

Plausible directions: a per-partition provenance marker outside the cursor, or
refusing to narrow at all for a dataset whose declared chain has changed since
its partitions were written (which needs the chain, or a fingerprint of it,
recorded somewhere durable).

Found during incremental review of PR #2330:
https://github.com/hyparam/hypaware/pull/2330#issuecomment-5903494330

Dominant language
JavaScript
Stars
11
Forks
0
Avg merge
2h 55m
Merged PRs (30d)
294

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hyparam/hypaware

All issues in hyparam/hypaware

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.