Cache reconcile can skip retirement when a dataset's cachePartitioning chain changes, leaving duplicate rows

Aperta
  • Qualcosa non va?
#2,332 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
javascript
Ambito
backend, databases

Direzione di ricerca

Start with src/core/cache/reconcile.js, especially partitionsOutsideScope, and read the linked PR #2330 discussion to understand the existing guard. Reproduce the chain-change scenario using appendRows and flushAll, as described in the issue. Done means reconciliation can no longer leave duplicate live rows when a partition’s label was written under a different cachePartitioning chain.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

partitionsOutsideScope in src/core/cache/reconcile.js decides a partition
provably holds none of a scope's rows by reading its source=<value> segment
and comparing it to the value the scope resolves to. The comparison is only
sound while the resolver that labelled the partition agrees with the one the
scope uses now.

Nothing on disk records which resolver wrote a label. The guard added in
PR #2330 checks declaration.source.columns[0] === 'client_name' against the
declaration in force at reconcile time, but the label was written by
whatever chain was in force at flush time. When a dataset's
cachePartitioning chain changes, old partitions keep labels the current
resolver never produces, and their scope-matching rows are skipped instead of
retired: two live rows for one identity, and silently wrong query results.

Reproduction

Driven entirely through production writers (appendRows + flushAll), never
appendRowsToPartition:

  1. Flush with source.columns = ['provider', 'client_name']. A row
    {client_name: 'hermes', session_id: 'hermes-s', part_id: 'p1'} with
    provider: 'openai' lands in source=openai.
  2. Corrupt that partition's cursor.json (the #2282 condition).
  3. Change the chain to the shipped ['client_name', 'conversation_source', 'provider'] (or remove the declaration), then reconcile the same scope.
reconcile returned 1 (did NOT refuse)
dirs on disk: ["_hypaware_spool","source=hermes","source=openai"]

ROWS AFTER CURSOR REPAIR:
    source=hermes -> new   (part_id=p1, session_id=hermes-s, client_name=hermes)
    source=openai -> stale (part_id=p1, session_id=hermes-s, client_name=hermes)

==> 2 live rows for the SAME identity (part_id=p1, session_id=hermes-s)

Controls: with no drift, both the shipped chain and the provider-first chain
correctly refuse.

Why it is not reachable today

ai-gateway/src/dataset.js:281 is the only cachePartitioning declaration in
the tree and has always led with client_name, so no partition on disk can
carry a disagreeing label. cachePartitioning is public plugin surface, so
this is a trap for a future declaration rather than a live defect, which is
why PR #2330 documents the precondition rather than closing it.

The trap worth naming

The reconcile-time guard tells a plugin author that a chain leading with
client_name is the supported shape. Correcting a chain to lead with
client_name is therefore the natural remediation, and it is exactly the
change that opens this gap on the rows already written under the old chain.

Why there is no small fix
  • Recording the resolved source in cursor.json does not work: this path
    skips a partition precisely because its cursor is unreadable, so anything
    stored there is unavailable when it is needed.
  • Reading the partition's rows to check is what the refusal exists to avoid
    (the live generation is unknown).
  • No predicate over the label alone can distinguish a label written by an
    agreeing resolver from one written by a disagreeing one.

Plausible directions: a per-partition provenance marker outside the cursor, or
refusing to narrow at all for a dataset whose declared chain has changed since
its partitions were written (which needs the chain, or a fingerprint of it,
recorded somewhere durable).

Found during incremental review of PR #2330:
https://github.com/hyparam/hypaware/pull/2330#issuecomment-5903494330

Lingua principale
JavaScript
Stelle
11
Fork
0
Merge medio
2h 55m
PR unite (30g)
294

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di hyparam/hypaware

Tutte le issue di hyparam/hypaware

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.