bug: legacy-layout FTS index returns no rows or errors after deferred-remap compaction

Aperta
#10,718 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
databases, search

Direzione di ricerca

Start with the legacy posting and document-set paths named in the issue: rust/lance-index/src/scalar/inverted/index/posting_reader.rs and doc_set.rs; then trace how inverted_index.rs passes the remapper. Reproduce with test_data/0.27.0/legacy_fts_index by deleting the named row, compacting with defer_index_remap: true, and running full text search. Done means the listed queries return the pre-compaction documents with identical per-document scores, covered by a regression test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug critical-fix performance

A full-text index in the layout Lance wrote before v0.28.0 returns no rows, or fails, after rows are deleted and the table is compacted with defer_index_remap: true under a legacy (v0) fragment reuse history. Queries then read the index through the history's row-address translation.

Reproduce

Use the checked-in fixture test_data/0.27.0/legacy_fts_index: 1 fragment, 3 rows, FTS index on text.

  1. Delete text = 'frodo was a puppy'.
  2. Run compact_files with defer_index_remap: true. This writes an FRI with index_version 0, keeps the same index uuid, and rewrites the fragments.
  3. Run full_text_search. These are the _score values per document:
query before compaction after
puppy "happy puppy" 0.13703518, tail doc 0.11839563 error: global BM25 query weight for token "puppy" must be finite and non-negative, got -0.15415066
happy "happy puppy" 1.0065652 0 rows
tail tail doc 0.86965203 0 rows
frodo puppy 0.27407035 / 0.23679125 error (same, for "frodo")

Compaction alone does not change BM25 inputs (global stats are not updated on delete or remap), so each surviving document should score exactly as before.

Cause

  1. Postings are never translated. Legacy postings are plain lists of row addresses (posting_batch_legacy, rust/lance-index/src/scalar/inverted/index/posting_reader.rs:549; live_posting_rows, rust/lance-index/src/scalar/inverted/index.rs:149). The remapper passed to load_legacy_index (inverted_index.rs:312) is only applied to the doc set. After compaction the postings point at addresses that no longer exist, so terms match nothing.
  2. Document lengths are misaligned. In DocSet::from_columns_with_doc_indices (doc_set.rs:588), the legacy branch runs filter_map(remap) over the row ids before zip(num_tokens). A deleted row is dropped, and every later document takes its predecessor's length. The document count and total length also shrink below what the postings and global stats assume. That is how a term's document frequency can exceed the document count and give a negative IDF.

Fix (verified on a branch)

  1. Keep deleted documents as tombstones. In the legacy branch of from_columns_with_doc_indices, map a deleted row to RowAddress::TOMBSTONE_ROW instead of dropping it. Pair it with its length before sorting, so the document count and total length match the untranslated index.
  2. Translate legacy postings at read time. When loading a legacy index with a remapper, read the doc row ids once and translate them. Keep sorted (built_with, translated) address pairs on the legacy posting metadata. At the end of posting-list loading, map each address through those pairs, drop deleted ones, and re-sort frequencies and positions together. Leave cached postings untranslated, so cache entries are never tied to one remap.

With both changes, every query above returns the pre-compaction rows with exactly the pre-compaction scores. A regression test should use the 0.27.0 fixture and the steps above, and assert per-document scores keyed by text.

Known remaining gap after this fix: if two or more docs of a translated legacy segment are deleted, the per-row counts combined_fields gets from row_stats_for_terms leave out the deleted rows (before compaction they include them). Single-field BM25 is unaffected.

Found while working on #10697.

Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.