Complete index types, build controls and index introspection

Aperta
#110 11 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, rust

Direzione di ricerca

Start by mapping each operation to its public Rust method, type, and validation path at the pinned core revision 40f15ed1b71b66950961b175829fdf17a763f108; use the selected Rust core as the semantic reference. Review the existing C/C++ contracts and normal entry point tests, plus the linked work in #101, #55, and #153. Done means the remaining index and introspection capabilities have explicit compatibility semantics and coverage for build/query behavior, error and empty cases, and persisted/reopened data, with exact commands and validation limits recorded.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Current enums expose common vector and scalar indices, while pylance supports additional families and richer lifecycle introspection.

Required capabilities

  • Add native NGram, ZoneMap and BloomFilter scalar-index paths, delivered in #138.
  • Add supported RTree and MinHash LSH scalar-index paths and complete the family/FTS naming audit. MinHash LSH is delivered in #144. RTree #149 is deferred by the maintainer until an upstream release contains the confirmed reload fix, tracked in #153. The scalar-name audit records FM as native/PyO3-only rather than a target high-level scalar family.
  • Add IVF_RQ and missing vector build/tuning controls through additive parameters or APIs. IVF_RQ=107 and complete native vector build options are delivered in #151. Both staged constructors share the trained DOT/precomputed-PQ scope protection, while definition-only subsets retain native behavior. Standalone trainer sample-rate/max-iterations controls and search effort are delivered in #158.
  • Expose MinHash search with its documented distance semantics, delivered in #144.
  • Add describe/statistics APIs, IVF model/centroid access and prewarm position/segment controls, delivered in #158.
  • Cover index metadata merging and physical segment merging with explicit coverage and compatibility semantics. Native artifact merging and physical merge/commit/reopen are delivered in #158. Same-UUID metadata-file merge publication is delivered in #166 through caller-supplied coverage and the existing commit path. Its exact combined tree passed Kimi review, 710 ordinary Rust tests, all seven native consumers, static/shared CMake consumers and all seven hosted checks. C11/C++17 publish all three fragments, reopen the URI and verify 48 fixed phrase-query row IDs.
  • Keep create/query/segment/prewarm/listing behavior and C++ wrappers consistent for each family.

Reuse #101 for its accepted segment-listing slice and #55 for distributed build coordination. Verify build/query results for each family, NULL/empty/error paths and persisted/reopened data. Version-specific training or optimizer changes depend on the core-baseline task.

Parent goal is #99. The source reference is pylance v14.0.2 at 40f15ed1b71b66950961b175829fdf17a763f108. Keep C/C++ contracts and meaningful normal-entry-point tests together. Record the exact source revision, commands, results, and validation limits.

Rust authority

The functional inventory uses pylance as a reference. The selected Lance Rust core at 40f15ed1b71b66950961b175829fdf17a763f108 defines available capabilities, parameter meanings, operation semantics and boundaries. C/C++ additions provide the foreign-language representation, ownership and error conversion needed to call that core.

Before implementation, map each operation to its public Rust method/type and validation path. Use direct Rust behavior as the semantic oracle. Any missing core operation is a core prerequisite. Binding-only product capabilities, policies and new query/retention semantics are outside this task. Existing public compatibility constraints must be handled explicitly during migration.

Rust operation limits

Dataset::merge_index_metadata supports Inverted in the target core. Other index families use merge_existing_index_segments and commit_existing_index_segments as applicable. Keep their distinct coverage, compatibility and error contracts. RTree depends on the core geo feature. Index type definitions and scalar parameters also live in lance-index-core at this baseline.

RTree release prerequisite

PR #149 is on hold by maintainer decision. #153 tracks the selected core's confirmed RTree reload false-negative bug, the merged upstream fix and the official-release adoption checks. Retain the current pin until that dependency is accepted, then rerun the preserved C/native indexed-versus-flat regression before resuming the PR. Other index-family and introspection work continues.

Family-route acceptance audit (2026-11-11)

The source/evidence audit at 27b2754652e886dc955502c18c02bf5517dbb8e0 found no functional defect in the non-held families. Existing shared create/query/segment/prewarm/listing routes, independent Rust semantic tests and real C11/C++17 consumers support the delivered checkboxes. This does not require a Cartesian product of every family, language and entry point. Retained #158/#166/#192/#198 execution evidence was matched to the relevant source; no new execution is claimed for the audit.

Native distinctions remain part of the contract: NGram/ZoneMap/BloomFilter prewarm is a no-op, so successful calls do not prove warmed cache entries. Their explicit scalar-segment paths fall back over the selected fragment domain. MinHash scalar-segment querying is rejected by native semantics. Metadata-file merging is Inverted-only; physical segment merge/commit remains a distinct operation. These are not missing binding accelerators.

The focused IVF_RQ/DOT coverage follow-up is accepted in #205, merged as aef913aa8d43e14bfa38889b1b5a4d2f7c206fe8. The C-ABI regression creates the index, queries it, closes/reopens the dataset, asserts independently calculated IDs and DOT distances, and compares with a native scanner on the persisted index. Both C queries assert that index partitions were loaded. Both partitions and all 64 candidate rows are refined, making this a deterministic DOT-route/persistence check rather than an approximate-recall benchmark. Existing L2/default coverage remains intact.

The final family-consistency checkbox remains open for the held RTree scope. Non-held family acceptance now includes the focused DOT follow-up. RTree #149/#153 is still excluded from delivered coverage, and FM retains the accepted native/PyO3-only inventory decision. The original issue scope is unchanged.

After the audit snapshot, ordinary index progress merged in #200 (8ba86c39c3fe83ff771df0aeab2fb21066e8930d) with separate native/consumer acceptance. That delivery closes #199; #205 subsequently resolves the DOT coverage follow-up. Neither lifts the RTree hold.

DOT follow-up execution and review

At 465742cf73b7b8ec34c415f976f26398e2844f80 on NieR with Rust 1.94.0, fmt, all-target check and Clippy passed. The exact DOT test passed (1 selected); the vector-index group passed (16 selected, including DOT); the legacy RQ default-bits test passed (1 selected). Independent source review found no blocker. The incoming #204 base delta does not change the index/scanner paths, test helper, dependencies or build configuration. Hosted Linux, MSRV, Clippy, format and Rustdoc passed; macOS and the additional review check were still pending when merged and are not claimed as successful execution. No new full native-consumer sweep is claimed for this test-only change.

Lingua principale
Rust
Stelle
8
Fork
17
Merge medio
1h 14m
PR unite (30g)
89

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance-c

Tutte le issue di lance-format/lance-c

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.