[Bug] Primary-key vector search full/detail modes can surface superseded rows for first-row tables without deletion vectors

Aperta
#9,895 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java
Ambito
databases, search

Direzione di ricerca

Reproduce the first-row primary-key table scenario with deletion vectors disabled and inspect PrimaryKeyVectorBucketSearch, PrimaryKeyIndexedSplitRead, and RawFileSplitRead. Trace how exact-scan positions are read without merge-on-read deduplication, then review the surrounding vector-search tests and related issue #9894. Done means full and detail searches no longer return superseded rows.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Search before asking
  • I searched in the issues and found nothing similar.
Paimon version

master

Compute Engine

Any (primary-key table vector search)

Minimal reproduce step

On a first-row merge-engine table with deletion vectors disabled and a pk-vector index, write a key, let it compact, then write the same key again (a version first-row drops on read). Run a vector search with vector-index.search-mode set to full or detail.

What doesn't meet your expectations?

The vector search can return the superseded row version, which a normal merge-on-read query would never return.

Root cause: the exact scan in PrimaryKeyVectorBucketSearch returns physical row positions per file with no dedup by primary key, and the readback (PrimaryKeyIndexedSplitRead / RawFileSplitRead) reads those positions directly without a merge-on-read pass. first-row resolves duplicate keys only through the merge/compaction path, so when a superseded version has not yet been merged away it is physically present and the exact scan can surface it. This is specific to the first-row engine with deletion vectors disabled; deletion-vector-enabled tables mark superseded rows and exclude them from the exact scan.

This predates #9894. That change extends the exact scan to level-0 APPEND files, which makes the surfacing more frequent (a fresh superseded write is the common case), but the underlying gap already applied to superseded versions promoted into compacted files.

Anything else?

One direction would be to apply a primary-key dedup to the exact-scan candidates before ranking (or require deletion vectors for full/detail on primary-key vector search). Filed for tracking, not blocking #9894.

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lingua principale
Java
Stelle
3.4k
Fork
1.4k
Merge medio
1g 14h
PR unite (30g)
468

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/paimon

Tutte le issue di apache/paimon

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.