feature: preserve matched element identity in vector search over list<struct>
メンテナーはふだん 2 日以内に返信
まだ誰も着手していません。
評価
調査の方向性
This is design work spanning lance-index scalar and vector code (rust/lance-index/src/scalar/list_element.rs and rust/lance-index/src/vector/transform.rs), with FTS _doc_index traversal as precedent. Start by reading how multivector flattening repeats parent IDs in transform.rs and how list_element.rs returns parent row-address sets. Done means the example returns B/0 for top-1 with coordinates carried through exact search, per the acceptance criteria.
索引モデルが issue の本文から書いたものです。
説明
Summary
Support element-level vector search over a list<struct<..., embedding: fixed_size_list<float>>> column, retaining the matched element's physical position and applying scalar predicates to that same element before ranking.
This enables retrieving a particular chunk, detection or clip together with its parent, vector distance and matching metadata. Milvus documents this as StructArray element-level vector search with element filtering.
Existing building blocks and remaining gap
- #10901 / #10950 implement
any_elementand its experimental scalar index. The tuple keys preserve same-element scalar correlation, but the postings and search results identify parent rows. - #11010 / #11062 add fragment-reuse remapping of those parent row addresses and declarations of nested schema-field dependencies. A schema field ID identifies a field such as
label; it does not identify one element within a particular row's list. - Existing multivector flattening repeats the parent row ID for each vector. It does not emit an element ordinal. Parent-level late-interaction retrieval is already supported, including WARP (#10927).
- FTS already has list-element document granularity and
_doc_indexcoordinates (#7788), providing a useful traversal and result-identity precedent.
Checked against main e602c45026ecace05081c880906615e6341ab713 and #11062 head dd4f15c576ef50ce55ca1d95bcc44b2a13d4271b:
- Scalar training schema: encoded key + parent
_rowaddr. - Scalar search returns parent row-address sets.
- Vector flattening repeats parent IDs.
Correctness example
For cosine query [1, 0], search detections satisfying label = 'car' AND confidence > 0.9:
| Parent | Element ordinal | Label | Confidence | Embedding |
|---|---|---|---|---|
| A | 0 | car | 0.95 | [0, 1] |
| A | 1 | person | 0.99 | [1, 0] |
| B | 0 | car | 0.96 | [0.8, 0.6] |
The best eligible hit is B, element 0, with cosine distance 0.2. A parent-level any_element prefilter admits A, but searching all its vectors lets its person vector win with distance 0. Child eligibility must reach vector scoring and candidate selection.
This is a desired feature contract; the table is a semantic example, not a claim that the nested vector API already accepts this query.
Proposed initial scope
- Resolve a vector leaf across one list-of-struct boundary, reusing/extracting the schema-derived traversal and physical-coordinate conventions already used by FTS.
- Carry
(parent identity, element coordinate)through exact search and an initial ANN path such as IVF_FLAT. Retain coordinates through flattening, shuffle/partition storage, filtering, candidate selection, refinement and result materialization. - Apply parent eligibility and same-element scalar predicates before selecting top-k. The current
listelementindex can supply coarse parent pruning; scalar-leaf scans and exact element rechecks can provide the initial child mask. - Return the parent identifier, matched coordinate, distance and metadata from the scored child. Preserve multiple hits from one parent unless grouping is explicitly requested.
- Define index metadata/routing so element search selects a coordinate-bearing representation or a correct exact fallback. Preserve existing parent-search defaults.
Coordinates are physical list positions in a pinned dataset snapshot. Filtering or skipping null/invalid elements must preserve those positions. An application-supplied child ID can provide durable identity across list edits.
Acceptance criteria
- The example above returns B/0 for top-1; scalar predicates and vector scoring always refer to the same child.
- Duplicate vectors/keys in different list positions remain distinguishable. Returned child metadata matches the scored coordinate.
- Exact, indexed, mixed indexed/unindexed and refined execution use the same coordinate and eligibility semantics; exhaustive unquantized IVF search agrees with the exact reference, and ANN recall is measured against it.
- Cover empty/null lists, null structs/vectors, sliced offsets, multiple fragments, deletes, list rewrites/overlays, compaction/remapping and historical snapshots, including stable-row-ID/FRI paths where supported.
- Field renames follow schema identity; dropping/casting indexed leaves invalidates the affected representation appropriately.
Related: #10900, #10950, #11010, #11062, #10927, #7788.
- 主要言語
- Rust
- スター
- 7.3k
- フォーク
- 929
- 平均マージ
- 2日 22時間
- マージ済み PR(30日)
- 572
環境構築
このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
lance-format/lance のほかの issue
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indices対応中かも @Borg93 が今日担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
lance-format/lance#11117 ·
メンテナーはふだん 2 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
lance-format/lance#10782 ·
メンテナーはふだん 2 日以内に返信
-
Upgrade DataFusion for the unparser stack-overflow fix再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
lance-format/lance#8225 ·
メンテナーはふだん 2 日以内に返信
-
enhancement
難易度 1/5 1時間未満 初心者へのやさしさ 84/100
lance-format/lance#6425 · コメント 2 件 ·
メンテナーはふだん 2 日以内に返信
-
Expose compute_pq_distance再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープンenhancement
難易度 1/5 1〜3時間 初心者へのやさしさ 62/100
lance-format/lance#5675 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
lance-format/lance の issue をすべて見る
似ている issue
-
area:cli area:security bug priority:medium
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
status: new topic: ga4gh-sdk type: bug
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
メンテナーはふだん 1 日以内に返信
-
comp-rbac
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
ClickHouse/ClickHouse#131875 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100
googleworkspace/cli#984 ·
メンテナーはふだん 1 日以内に返信