Complete index types, build controls and index introspection
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 25/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Ambito
- backend-api-design, database
Direzione di ricerca
Start by mapping each operation to its public Rust method, type, and validation path at the pinned core revision 40f15ed1b71b66950961b175829fdf17a763f108; use the selected Rust core as the semantic reference. Review the existing C/C++ contracts and normal entry point tests, plus the linked work in #101, #55, and #153. Done means the remaining index and introspection capabilities have explicit compatibility semantics and coverage for build/query behavior, error and empty cases, and persisted/reopened data, with exact commands and validation limits recorded.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Current enums expose common vector and scalar indices, while pylance supports additional families and richer lifecycle introspection.
Required capabilities
- Add native NGram, ZoneMap and BloomFilter scalar-index paths, delivered in #138.
- Add supported RTree and MinHash LSH scalar-index paths and complete the family/FTS naming audit. MinHash LSH is delivered in #144. RTree #149 is deferred by the maintainer until an upstream release contains the confirmed reload fix, tracked in #153. The scalar-name audit records FM as native/PyO3-only rather than a target high-level scalar family.
- Add IVF_RQ and missing vector build/tuning controls through additive parameters or APIs. IVF_RQ=107 and complete native vector build options are delivered in #151. Both staged constructors share the trained DOT/precomputed-PQ scope protection, while definition-only subsets retain native behavior. Standalone trainer sample-rate/max-iterations controls and search effort are delivered in #158.
- Expose MinHash search with its documented distance semantics, delivered in #144.
- Add describe/statistics APIs, IVF model/centroid access and prewarm position/segment controls, delivered in #158.
- Cover index metadata merging and physical segment merging with explicit coverage and compatibility semantics. Native artifact merging and physical merge/commit/reopen are delivered in #158. Same-UUID metadata-file merge publication is delivered in #166 through caller-supplied coverage and the existing commit path. Its exact combined tree passed Kimi review, 710 ordinary Rust tests, all seven native consumers, static/shared CMake consumers and all seven hosted checks. C11/C++17 publish all three fragments, reopen the URI and verify 48 fixed phrase-query row IDs.
- Keep create/query/segment/prewarm/listing behavior and C++ wrappers consistent for each family.
Reuse #101 for its accepted segment-listing slice and #55 for distributed build coordination. Verify build/query results for each family, NULL/empty/error paths and persisted/reopened data. Version-specific training or optimizer changes depend on the core-baseline task.
Parent goal is #99. The source reference is pylance v14.0.2 at 40f15ed1b71b66950961b175829fdf17a763f108. Keep C/C++ contracts and meaningful normal-entry-point tests together. Record the exact source revision, commands, results, and validation limits.
Rust authority
The functional inventory uses pylance as a reference. The selected Lance Rust core at 40f15ed1b71b66950961b175829fdf17a763f108 defines available capabilities, parameter meanings, operation semantics and boundaries. C/C++ additions provide the foreign-language representation, ownership and error conversion needed to call that core.
Before implementation, map each operation to its public Rust method/type and validation path. Use direct Rust behavior as the semantic oracle. Any missing core operation is a core prerequisite. Binding-only product capabilities, policies and new query/retention semantics are outside this task. Existing public compatibility constraints must be handled explicitly during migration.
Rust operation limits
Dataset::merge_index_metadata supports Inverted in the target core. Other index families use merge_existing_index_segments and commit_existing_index_segments as applicable. Keep their distinct coverage, compatibility and error contracts. RTree depends on the core geo feature. Index type definitions and scalar parameters also live in lance-index-core at this baseline.
RTree release prerequisite
PR #149 is on hold by maintainer decision. #153 tracks the selected core's confirmed RTree reload false-negative bug, the merged upstream fix and the official-release adoption checks. Retain the current pin until that dependency is accepted, then rerun the preserved C/native indexed-versus-flat regression before resuming the PR. Other index-family and introspection work continues.
Family-route acceptance audit (2026-11-11)
The source/evidence audit at 27b2754652e886dc955502c18c02bf5517dbb8e0 found no functional defect in the non-held families. Existing shared create/query/segment/prewarm/listing routes, independent Rust semantic tests and real C11/C++17 consumers support the delivered checkboxes. This does not require a Cartesian product of every family, language and entry point. Retained #158/#166/#192/#198 execution evidence was matched to the relevant source; no new execution is claimed for the audit.
Native distinctions remain part of the contract: NGram/ZoneMap/BloomFilter prewarm is a no-op, so successful calls do not prove warmed cache entries. Their explicit scalar-segment paths fall back over the selected fragment domain. MinHash scalar-segment querying is rejected by native semantics. Metadata-file merging is Inverted-only; physical segment merge/commit remains a distinct operation. These are not missing binding accelerators.
The focused IVF_RQ/DOT coverage follow-up is accepted in #205, merged as aef913aa8d43e14bfa38889b1b5a4d2f7c206fe8. The C-ABI regression creates the index, queries it, closes/reopens the dataset, asserts independently calculated IDs and DOT distances, and compares with a native scanner on the persisted index. Both C queries assert that index partitions were loaded. Both partitions and all 64 candidate rows are refined, making this a deterministic DOT-route/persistence check rather than an approximate-recall benchmark. Existing L2/default coverage remains intact.
The final family-consistency checkbox remains open for the held RTree scope. Non-held family acceptance now includes the focused DOT follow-up. RTree #149/#153 is still excluded from delivered coverage, and FM retains the accepted native/PyO3-only inventory decision. The original issue scope is unchanged.
After the audit snapshot, ordinary index progress merged in #200 (8ba86c39c3fe83ff771df0aeab2fb21066e8930d) with separate native/consumer acceptance. That delivery closes #199; #205 subsequently resolves the DOT coverage follow-up. Neither lifts the RTree hold.
DOT follow-up execution and review
At 465742cf73b7b8ec34c415f976f26398e2844f80 on NieR with Rust 1.94.0, fmt, all-target check and Clippy passed. The exact DOT test passed (1 selected); the vector-index group passed (16 selected, including DOT); the legacy RQ default-bits test passed (1 selected). Independent source review found no blocker. The incoming #204 base delta does not change the index/scanner paths, test helper, dependencies or build configuration. Hosted Linux, MSRV, Clippy, format and Rustdoc passed; macOS and the additional review check were still pending when merged and are not claimed as successful execution. No new full native-consumer sweep is claimed for this test-only change.
- Lingua principale
- Rust
- Stelle
- 8
- Fork
- 17
- Merge medio
- 1h 14m
- PR unite (30g)
- 89
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di lance-format/lance-c
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 15/100
lance-format/lance-c#153 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
lance-format/lance-c#103 · 4 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 20/100
lance-format/lance-c#102 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 15/100
lance-format/lance-c#99 · 8 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di lance-format/lance-c
Issue simili
-
comp-rbac
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
ClickHouse/ClickHouse#131875 ·
I maintainer di solito rispondono entro 1 giorno
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesForse già presa @Borg93 l’ha presa oggi. Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
lance-format/lance#11117 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
storytold/photocraft#15232 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Forse già presa @Ynyesto l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
foundry-rs/foundry#18485 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
bondybondbond/rushcut#238 · 1 commento ·