feature: preserve matched element identity in vector search over list<struct>
I maintainer di solito rispondono entro 2 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 8/100
Direzione di ricerca
This is design work spanning lance-index scalar and vector code (rust/lance-index/src/scalar/list_element.rs and rust/lance-index/src/vector/transform.rs), with FTS _doc_index traversal as precedent. Start by reading how multivector flattening repeats parent IDs in transform.rs and how list_element.rs returns parent row-address sets. Done means the example returns B/0 for top-1 with coordinates carried through exact search, per the acceptance criteria.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
Support element-level vector search over a list<struct<..., embedding: fixed_size_list<float>>> column, retaining the matched element's physical position and applying scalar predicates to that same element before ranking.
This enables retrieving a particular chunk, detection or clip together with its parent, vector distance and matching metadata. Milvus documents this as StructArray element-level vector search with element filtering.
Existing building blocks and remaining gap
- #10901 / #10950 implement
any_elementand its experimental scalar index. The tuple keys preserve same-element scalar correlation, but the postings and search results identify parent rows. - #11010 / #11062 add fragment-reuse remapping of those parent row addresses and declarations of nested schema-field dependencies. A schema field ID identifies a field such as
label; it does not identify one element within a particular row's list. - Existing multivector flattening repeats the parent row ID for each vector. It does not emit an element ordinal. Parent-level late-interaction retrieval is already supported, including WARP (#10927).
- FTS already has list-element document granularity and
_doc_indexcoordinates (#7788), providing a useful traversal and result-identity precedent.
Checked against main e602c45026ecace05081c880906615e6341ab713 and #11062 head dd4f15c576ef50ce55ca1d95bcc44b2a13d4271b:
- Scalar training schema: encoded key + parent
_rowaddr. - Scalar search returns parent row-address sets.
- Vector flattening repeats parent IDs.
Correctness example
For cosine query [1, 0], search detections satisfying label = 'car' AND confidence > 0.9:
| Parent | Element ordinal | Label | Confidence | Embedding |
|---|---|---|---|---|
| A | 0 | car | 0.95 | [0, 1] |
| A | 1 | person | 0.99 | [1, 0] |
| B | 0 | car | 0.96 | [0.8, 0.6] |
The best eligible hit is B, element 0, with cosine distance 0.2. A parent-level any_element prefilter admits A, but searching all its vectors lets its person vector win with distance 0. Child eligibility must reach vector scoring and candidate selection.
This is a desired feature contract; the table is a semantic example, not a claim that the nested vector API already accepts this query.
Proposed initial scope
- Resolve a vector leaf across one list-of-struct boundary, reusing/extracting the schema-derived traversal and physical-coordinate conventions already used by FTS.
- Carry
(parent identity, element coordinate)through exact search and an initial ANN path such as IVF_FLAT. Retain coordinates through flattening, shuffle/partition storage, filtering, candidate selection, refinement and result materialization. - Apply parent eligibility and same-element scalar predicates before selecting top-k. The current
listelementindex can supply coarse parent pruning; scalar-leaf scans and exact element rechecks can provide the initial child mask. - Return the parent identifier, matched coordinate, distance and metadata from the scored child. Preserve multiple hits from one parent unless grouping is explicitly requested.
- Define index metadata/routing so element search selects a coordinate-bearing representation or a correct exact fallback. Preserve existing parent-search defaults.
Coordinates are physical list positions in a pinned dataset snapshot. Filtering or skipping null/invalid elements must preserve those positions. An application-supplied child ID can provide durable identity across list edits.
Acceptance criteria
- The example above returns B/0 for top-1; scalar predicates and vector scoring always refer to the same child.
- Duplicate vectors/keys in different list positions remain distinguishable. Returned child metadata matches the scored coordinate.
- Exact, indexed, mixed indexed/unindexed and refined execution use the same coordinate and eligibility semantics; exhaustive unquantized IVF search agrees with the exact reference, and ANN recall is measured against it.
- Cover empty/null lists, null structs/vectors, sliced offsets, multiple fragments, deletes, list rewrites/overlays, compaction/remapping and historical snapshots, including stable-row-ID/FRI paths where supported.
- Field renames follow schema identity; dropping/casting indexed leaves invalidates the affected representation appropriately.
Related: #10900, #10950, #11010, #11062, #10927, #7788.
- Lingua principale
- Rust
- Stelle
- 7.3k
- Fork
- 929
- Merge medio
- 2g 22h
- PR unite (30g)
- 572
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di lance-format/lance
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesForse già presa @Borg93 l’ha presa oggi. Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#11117 ·
I maintainer di solito rispondono entro 2 giorni
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#10782 ·
I maintainer di solito rispondono entro 2 giorni
-
Upgrade DataFusion for the unparser stack-overflow fixForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
lance-format/lance#8225 ·
I maintainer di solito rispondono entro 2 giorni
-
enhancement
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 84/100
lance-format/lance#6425 · 2 commenti ·
I maintainer di solito rispondono entro 2 giorni
-
Expose compute_pq_distanceForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Apertaenhancement
Difficoltà 1/5 1-3 ore Idoneità per principianti 62/100
lance-format/lance#5675 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
Tutte le issue di lance-format/lance
Issue simili
-
comp-rbac
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
ClickHouse/ClickHouse#131875 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Forse già presa @Ynyesto l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
foundry-rs/foundry#18485 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
bondybondbond/rushcut#238 · 1 commento ·
-
area: app type: bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno