Refactor and Split `rust/lance/src/index.rs` into submodules

Start from rust/lance/src/index.rs and the line ranges in the issue, then check how vector/hamming.rs and vector/details.rs reach private items through crate::index. Done means the split files compile, every existing path still resolves through re-exports, and the test-name list and attribute counts match before and after. Confirm this with cargo test -p lance --lib -- index:: and the diff of the test-name lists.

This is a pure move: code is relocated into submodules, no logic changes, 100% path compatibility through re-exports. The mechanics follow PR #9989 (dataset/blob.rs split). Line numbers below refer to the commit above.

1. Current module

Role. The crate::index facade.

Existing children and re-exports

• 12 children (81–92): api, append, create, frag_reuse, frag_reuse_reader, frag_reuse_remapping, frag_reuse_with_stable_row_ids, mem_wal, prefilter, scalar, scalar_logical, vector.
• Re-exports: IndexParams, IndexBuildProgress, NoopIndexBuildProgress, DatasetIndexExt, IndexSegment, IntoIndexSegment, FilterLoader, PreFilter, LogicalIvfView, LogicalVectorIndex, CreateIndexBuilder, IndexDescription, pub(crate) vector_index_details*.

Concern groups (its own code)

• 118–151, 912–1187, 1505–1710 segment metadata validation/replay/type detection.
• 152–911 segment coverage & provenance: remap_merged_segment_coverage, History, Snapshot, fresh_fragments, prune_stale_segment_coverage.
• 1188–1504 prewarm.
• 1711–1874 cache keys.
• 1875–1922 misc.
• 1923–1930 unused pub trait IndexBuilder.
• 1931–2150 remap.
• 2151–2197 ScalarIndexInfo.
• 2198–2211 open_index_proto.
• 2212–2419 IndexDescriptionImpl.
• 2420–3517 impl DatasetIndexExt for Dataset (1,100 lines).
• 3518–3754 statistics.
• 3755–3932 listing/usability: load_all_indices, index_is_usable, unsupported_index_version, ….
• 3933–4043 pub trait DatasetIndexInternalExt.
• 4044–4538 open paths impl Dataset { *_with_purpose } (contains one 331-line fn).
• 4539–5077 impl DatasetIndexInternalExt for Dataset.
• 5078–5126 tail helpers.

Hidden coupling

• Descendants resolve private imports through the facade: INDEX_FILE_NAME from create.rs, frag_reuse_remapping.rs, ivf/v2.rs; IndexType from vector/hamming.rs; IndexMetadata from scalar.rs; pb from vector.rs.
• vector/hamming.rs:41 imports the private fn filter_index_segments_by_ids.

Consumers

• DatasetIndexExt/DatasetIndexInternalExt in ~100 files incl. python/java; load_all_indices ~199 references.
• From other modules: index_is_usable, remap_index, build_index_metadata_from_segments, validate_segment_params_compatible, segment_has_vector_details, index_type_is_known, IvfIndexStateCacheKey, open_index_proto.

Tests

• One mod tests: 143 fns; #[test] 1, #[tokio::test 142 (incl. one commented line at 8396), #[lance_test_macros::test(tokio::test)] 1, #[rstest 15, #[case 56, #[values 1; 4 #[cfg(feature = "geo")] tests.
• Shared fixtures: 5189–5280 and 5589–5775.
• Private items used: aggregate_fts_prewarm_results, OpenedSegmentPrewarmResult, History, remap_deletes_all_indexed_rows, IndexDescriptionImpl::try_new, resolve_index_column, sources_at_build.

2. Split

Keep index.rs as the facade. New names do not collide with the 12 existing children.

Facade re-exports (same visibility as today): pub use open::DatasetIndexInternalExt is not needed because the trait stays in the facade; pub(crate) use segments::{replay_staged_segments, build_index_metadata_from_segments, validate_segment_params_compatible, segment_has_vector_details, index_type_is_known};, pub(crate) use remap::remap_index;, pub(crate) use listing::{unsupported_index_version, index_is_usable, warn_about_unsupported_indices, load_all_indices, load_all_indices_by_name};, private use segments::filter_index_segments_by_ids; and use open::open_index_proto; (descendants vector/hamming.rs, vector/details.rs resolve them through crate::index::). Register the six new test files in dataset/tests/mod.rs.

3. Cautions

• The items re-exported with pub(crate) use must be declared pub in their child (lint); filter_index_segments_by_ids and open_index_proto become pub(super) (visible to all of index's descendants, which is exactly the current reach).
• Private cross-file helpers to widen to pub(super): validate_segment_metadata, remap_merged_segment_coverage, prune_stale_segment_coverage, validate_segment_index_details, the ten segment_has_*_details, prewarm_index_segments_by_metadata, IndexDescriptionImpl + try_new, find_frag_reuse_index_meta, index_statistics_{frag_reuse,mem_wal,scalar}, resolve_index_column, mem_wal_catch_up_would_advance (if ext.rs and its caller separate; they do not: both stay in ext.rs), load_index_with_purpose and the other *_with_purpose methods stay pub(crate) methods (impl items are exempt).
• Keep in the facade every import that descendants resolve through crate::index:: / super::: INDEX_FILE_NAME, IndexType, IndexMetadata, pb (see §1); do not "clean up" unused-looking imports there until cargo check --all-targets passes.
• Moved bodies use bare child-module paths; add module-name imports instead of rewriting them: segment_coverage.rs use super::{append, frag_reuse}; (170, 235); prewarm.rs use super::scalar; (1274); remap.rs use super::frag_reuse; (1981); statistics.rs use super::scalar; (3680, 3685); ext.rs use super::{append, frag_reuse, frag_reuse_reader, vector}; (2700–3293); open.rs use super::{frag_reuse, scalar, vector}; (4063–5037).
• open_vector_index_from_metadata_with_plan is a 331-line function: it moves whole.
• #[cfg(feature = "geo")] blocks at 2851, 2931, 2936 travel with ext.rs; the four geo tests keep their attributes in dataset_index_segments.rs.
• Do not touch dataset/tests/dataset_index.rs (already 7,849 lines).
• Test invariant: 143 test fns; attribute counts as listed in §1 (note the commented // #[tokio::test] at 8396 inflates a raw grep by one).
• Delete rust/lance/src/index.rs from ci/module_size_exceptions.txt.
• Verify: cargo test -p lance --lib -- index:: dataset::tests::dataset_index_; python/java cargo check (both import DatasetIndexExt, IndexSegment, NoopIndexBuildProgress via lance::index::).

4. Verification

Run before and after the move (all from the repo root):

# every new file < 3,000 lines and < 400 KB; CI replay against main
wc -lc <facade.rs> <facade>/*.rs <facade>/tests/*.rs
python3 ci/check_module_size.py main HEAD
# test invariant: attribute counts and the multiset of test fn names are unchanged
grep -hoE '#\[(test|tokio::test|rstest|case|values|serial_test::serial|test_log::test|lance_test_macros::test)' <old file | new files> | sort | uniq -c
cargo test -p <crate> --lib -- --list 2>/dev/null | grep ': test$' | sed 's/.*:://' | sort > names.{before,after}.txt && diff names.before.txt names.after.txt
# exactly what CI runs
cargo clippy --locked --features "$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | sort -u | paste -s -d ,)" --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
cargo fmt --all -- --check
# module-specific commands: see the last bullet of section 3
(cd python && cargo check)          # package pylance
(cd java/lance-jni && cargo check)  # package lance-jni

Pure-move checklist for the PR description: no function body changed, no item renamed, no signature changed, visibility only widened where listed in section 3, every pre-existing path still resolves (re-exports), exceptions-list line removed (if listed), attribute counts and test names identical.