bug: legacy-layout FTS index returns no rows or errors after deferred-remap compaction
I maintainer di solito rispondono entro 2 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 38/100
Direzione di ricerca
Start with the legacy posting and document-set paths named in the issue: rust/lance-index/src/scalar/inverted/index/posting_reader.rs and doc_set.rs; then trace how inverted_index.rs passes the remapper. Reproduce with test_data/0.27.0/legacy_fts_index by deleting the named row, compacting with defer_index_remap: true, and running full text search. Done means the listed queries return the pre-compaction documents with identical per-document scores, covered by a regression test.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
A full-text index in the layout Lance wrote before v0.28.0 returns no rows, or fails, after rows are deleted and the table is compacted with defer_index_remap: true under a legacy (v0) fragment reuse history. Queries then read the index through the history's row-address translation.
Reproduce
Use the checked-in fixture test_data/0.27.0/legacy_fts_index: 1 fragment, 3 rows, FTS index on text.
- Delete
text = 'frodo was a puppy'. - Run
compact_fileswithdefer_index_remap: true. This writes an FRI withindex_version0, keeps the same index uuid, and rewrites the fragments. - Run
full_text_search. These are the_scorevalues per document:
| query | before compaction | after |
|---|---|---|
puppy |
"happy puppy" 0.13703518, tail doc 0.11839563 | error: global BM25 query weight for token "puppy" must be finite and non-negative, got -0.15415066 |
happy |
"happy puppy" 1.0065652 | 0 rows |
tail |
tail doc 0.86965203 | 0 rows |
frodo puppy |
0.27407035 / 0.23679125 | error (same, for "frodo") |
Compaction alone does not change BM25 inputs (global stats are not updated on delete or remap), so each surviving document should score exactly as before.
Cause
- Postings are never translated. Legacy postings are plain lists of row addresses (
posting_batch_legacy,rust/lance-index/src/scalar/inverted/index/posting_reader.rs:549;live_posting_rows,rust/lance-index/src/scalar/inverted/index.rs:149). The remapper passed toload_legacy_index(inverted_index.rs:312) is only applied to the doc set. After compaction the postings point at addresses that no longer exist, so terms match nothing. - Document lengths are misaligned. In
DocSet::from_columns_with_doc_indices(doc_set.rs:588), the legacy branch runsfilter_map(remap)over the row ids beforezip(num_tokens). A deleted row is dropped, and every later document takes its predecessor's length. The document count and total length also shrink below what the postings and global stats assume. That is how a term's document frequency can exceed the document count and give a negative IDF.
Fix (verified on a branch)
- Keep deleted documents as tombstones. In the legacy branch of
from_columns_with_doc_indices, map a deleted row toRowAddress::TOMBSTONE_ROWinstead of dropping it. Pair it with its length before sorting, so the document count and total length match the untranslated index. - Translate legacy postings at read time. When loading a legacy index with a remapper, read the doc row ids once and translate them. Keep sorted
(built_with, translated)address pairs on the legacy posting metadata. At the end of posting-list loading, map each address through those pairs, drop deleted ones, and re-sort frequencies and positions together. Leave cached postings untranslated, so cache entries are never tied to one remap.
With both changes, every query above returns the pre-compaction rows with exactly the pre-compaction scores. A regression test should use the 0.27.0 fixture and the steps above, and assert per-document scores keyed by text.
Known remaining gap after this fix: if two or more docs of a translated legacy segment are deleted, the per-row counts combined_fields gets from row_stats_for_terms leave out the deleted rows (before compaction they include them). Single-field BM25 is unaffected.
Found while working on #10697.
- Lingua principale
- Rust
- Stelle
- 7.3k
- Fork
- 929
- Merge medio
- 2g 22h
- PR unite (30g)
- 572
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di lance-format/lance
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesForse già presa @Borg93 l’ha presa oggi. Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#11117 ·
I maintainer di solito rispondono entro 2 giorni
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#10782 ·
I maintainer di solito rispondono entro 2 giorni
-
Upgrade DataFusion for the unparser stack-overflow fixForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
lance-format/lance#8225 ·
I maintainer di solito rispondono entro 2 giorni
-
enhancement
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 84/100
lance-format/lance#6425 · 2 commenti ·
I maintainer di solito rispondono entro 2 giorni
-
Expose compute_pq_distanceForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Apertaenhancement
Difficoltà 1/5 1-3 ore Idoneità per principianti 62/100
lance-format/lance#5675 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
Tutte le issue di lance-format/lance
Issue simili
-
comp-rbac
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
ClickHouse/ClickHouse#131875 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Forse già presa @Ynyesto l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
foundry-rs/foundry#18485 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
bondybondbond/rushcut#238 · 1 commento ·
-
area: app type: bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno