Complete index types, build controls and index introspection

Abierto
#110 11 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp, rust

Línea de trabajo

Start by mapping each operation to its public Rust method, type, and validation path at the pinned core revision 40f15ed1b71b66950961b175829fdf17a763f108; use the selected Rust core as the semantic reference. Review the existing C/C++ contracts and normal entry point tests, plus the linked work in #101, #55, and #153. Done means the remaining index and introspection capabilities have explicit compatibility semantics and coverage for build/query behavior, error and empty cases, and persisted/reopened data, with exact commands and validation limits recorded.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Current enums expose common vector and scalar indices, while pylance supports additional families and richer lifecycle introspection.

Required capabilities

  • Add native NGram, ZoneMap and BloomFilter scalar-index paths, delivered in #138.
  • Add supported RTree and MinHash LSH scalar-index paths and complete the family/FTS naming audit. MinHash LSH is delivered in #144. RTree #149 is deferred by the maintainer until an upstream release contains the confirmed reload fix, tracked in #153. The scalar-name audit records FM as native/PyO3-only rather than a target high-level scalar family.
  • Add IVF_RQ and missing vector build/tuning controls through additive parameters or APIs. IVF_RQ=107 and complete native vector build options are delivered in #151. Both staged constructors share the trained DOT/precomputed-PQ scope protection, while definition-only subsets retain native behavior. Standalone trainer sample-rate/max-iterations controls and search effort are delivered in #158.
  • Expose MinHash search with its documented distance semantics, delivered in #144.
  • Add describe/statistics APIs, IVF model/centroid access and prewarm position/segment controls, delivered in #158.
  • Cover index metadata merging and physical segment merging with explicit coverage and compatibility semantics. Native artifact merging and physical merge/commit/reopen are delivered in #158. Same-UUID metadata-file merge publication is delivered in #166 through caller-supplied coverage and the existing commit path. Its exact combined tree passed Kimi review, 710 ordinary Rust tests, all seven native consumers, static/shared CMake consumers and all seven hosted checks. C11/C++17 publish all three fragments, reopen the URI and verify 48 fixed phrase-query row IDs.
  • Keep create/query/segment/prewarm/listing behavior and C++ wrappers consistent for each family.

Reuse #101 for its accepted segment-listing slice and #55 for distributed build coordination. Verify build/query results for each family, NULL/empty/error paths and persisted/reopened data. Version-specific training or optimizer changes depend on the core-baseline task.

Parent goal is #99. The source reference is pylance v14.0.2 at 40f15ed1b71b66950961b175829fdf17a763f108. Keep C/C++ contracts and meaningful normal-entry-point tests together. Record the exact source revision, commands, results, and validation limits.

Rust authority

The functional inventory uses pylance as a reference. The selected Lance Rust core at 40f15ed1b71b66950961b175829fdf17a763f108 defines available capabilities, parameter meanings, operation semantics and boundaries. C/C++ additions provide the foreign-language representation, ownership and error conversion needed to call that core.

Before implementation, map each operation to its public Rust method/type and validation path. Use direct Rust behavior as the semantic oracle. Any missing core operation is a core prerequisite. Binding-only product capabilities, policies and new query/retention semantics are outside this task. Existing public compatibility constraints must be handled explicitly during migration.

Rust operation limits

Dataset::merge_index_metadata supports Inverted in the target core. Other index families use merge_existing_index_segments and commit_existing_index_segments as applicable. Keep their distinct coverage, compatibility and error contracts. RTree depends on the core geo feature. Index type definitions and scalar parameters also live in lance-index-core at this baseline.

RTree release prerequisite

PR #149 is on hold by maintainer decision. #153 tracks the selected core's confirmed RTree reload false-negative bug, the merged upstream fix and the official-release adoption checks. Retain the current pin until that dependency is accepted, then rerun the preserved C/native indexed-versus-flat regression before resuming the PR. Other index-family and introspection work continues.

Family-route acceptance audit (2026-11-11)

The source/evidence audit at 27b2754652e886dc955502c18c02bf5517dbb8e0 found no functional defect in the non-held families. Existing shared create/query/segment/prewarm/listing routes, independent Rust semantic tests and real C11/C++17 consumers support the delivered checkboxes. This does not require a Cartesian product of every family, language and entry point. Retained #158/#166/#192/#198 execution evidence was matched to the relevant source; no new execution is claimed for the audit.

Native distinctions remain part of the contract: NGram/ZoneMap/BloomFilter prewarm is a no-op, so successful calls do not prove warmed cache entries. Their explicit scalar-segment paths fall back over the selected fragment domain. MinHash scalar-segment querying is rejected by native semantics. Metadata-file merging is Inverted-only; physical segment merge/commit remains a distinct operation. These are not missing binding accelerators.

The focused IVF_RQ/DOT coverage follow-up is accepted in #205, merged as aef913aa8d43e14bfa38889b1b5a4d2f7c206fe8. The C-ABI regression creates the index, queries it, closes/reopens the dataset, asserts independently calculated IDs and DOT distances, and compares with a native scanner on the persisted index. Both C queries assert that index partitions were loaded. Both partitions and all 64 candidate rows are refined, making this a deterministic DOT-route/persistence check rather than an approximate-recall benchmark. Existing L2/default coverage remains intact.

The final family-consistency checkbox remains open for the held RTree scope. Non-held family acceptance now includes the focused DOT follow-up. RTree #149/#153 is still excluded from delivered coverage, and FM retains the accepted native/PyO3-only inventory decision. The original issue scope is unchanged.

After the audit snapshot, ordinary index progress merged in #200 (8ba86c39c3fe83ff771df0aeab2fb21066e8930d) with separate native/consumer acceptance. That delivery closes #199; #205 subsequently resolves the DOT coverage follow-up. Neither lifts the RTree hold.

DOT follow-up execution and review

At 465742cf73b7b8ec34c415f976f26398e2844f80 on NieR with Rust 1.94.0, fmt, all-target check and Clippy passed. The exact DOT test passed (1 selected); the vector-index group passed (16 selected, including DOT); the legacy RQ default-bits test passed (1 selected). Independent source review found no blocker. The incoming #204 base delta does not change the index/scanner paths, test helper, dependencies or build configuration. Hosted Linux, MSRV, Clippy, format and Rustdoc passed; macOS and the additional review check were still pending when merged and are not claimed as successful execution. No new full native-consumer sweep is claimed for this test-only change.

Lenguaje dominante
Rust
Estrellas
8
Forks
17
Merge medio
1 h 14 min
PR fusionados (30 d)
89

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance-c

Todos los issues de lance-format/lance-c

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.