Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Refactor and Split `rust/lance-index/src/scalar/inverted/wand.rs` into submodules

Open
#10,060 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
search

Research direction

Start with rust/lance-index/src/scalar/inverted/wand.rs, the facade that becomes the parent of the new wand/ directory, and the two #[path] files wand_maxscore.rs and wand_intersection.rs. Before moving anything, run cargo test -p lance-index --lib -- inverted::wand and save the sorted test names, since they must match afterwards. Done means a pure move with no body, name or signature changes, the test names and attribute counts unchanged, the index.rs glob exporting the same names, and the module-size checks passing.

Written by the indexing model from the issue text.

Description

This is a pure move: code is relocated into submodules, no logic changes, 100% path compatibility through re-exports. The mechanics follow PR #9989 (dataset/blob.rs split). Line numbers below refer to the commit above.

1. Current module

Role. WAND / MaxScore / bulk-AND search over posting lists.

Header and existing children

  • 4–5 #[cfg(test)] use std::cell::Cell;; imports 30–50.
  • 25–28 #[path = "wand_intersection.rs"] mod intersection; and #[path = "wand_maxscore.rs"] mod maxscore; (files live in inverted/).
  • wand_maxscore.rs (732 lines) already holds impl PostingIterator / impl Wand pieces with pub(super) methods and reads private fields of both structs.

Concern groups

  • 52–77 consts/types/norm cache.
  • 78–237 TopKCollector.
  • 238–333 tuning/env: pub static FLAT_SEARCH_PERCENT_THRESHOLD, BulkAndMode, CompetitiveFloorMode.
  • 334–403 SIMD find_next_geq*; 404–426 bounds; 427–463 pub(super) GroupedTermScorer.
  • 464–490 pub struct PostingIterator + CompressedState/BlockMaxScore; 807–1778 impl PostingIterator (972 lines); 1779–1822 validate_modern_posting_doc_ids.
  • 1823–1991 MaxScore helpers.
  • 1992–2305 document sources: DocCandidate, FlatDocuments, trait WandDocuments, ModernWandDocuments, LegacyWandDocuments, #[cfg(test)] impl WandDocuments for DocSet.
  • 2306–2421 head/tail postings.
  • 2422–2503 pub struct Wand (many #[cfg(test)] counter fields).
  • 2516–5725 impl Wand (3,210 lines):
    • 2520–2702 construction; 2704–2964 search/flat_search
    • 2966–3662 maxscore_search = one 697-line fn with a local macro_rules!
    • 3664–3866 score-first-AND helpers; 3868–4105 classic next
    • 4115–5023 and_bulk_search = one 909-line fn with a local macro_rules! and x86_64 cfg blocks
    • 5144–5639 OR head/tail; 5641–5723 phrase checks
  • 5726–6035 phrase scan + position cursors.
  • 6036–6370 pub(super) WandCursor + a second impl Wand.
  • 6371–6420 float helpers.

Visibility and cfg facts

  • pub(super) items: 11 top-level (429, 1782, 1831, 2005, 2007, 2033, 2046, 2087, 2176, 6041, 6381) and 20 methods.
  • pub(crate) methods at 957, 967, 1063, 1068, 2093, 2102, 2182, 2520, 2638, 2646, 2706.
  • 76 #[cfg(test)] instrumentation sites in non-test code; target_arch/target_feature at 334–397, 4407–4818.

Wiring

  • inverted.rs:23 mod wand; (private).

Consumers

  • compound.rs:33-36, cross_column.rs:34-36.
  • index.rs:72 use super::{…, wand::*}, whose glob flows into index/partition.rs, index/search.rs, index/search_candidates.rs, index/posting_reader.rs, index/tests/scoring.rs.

Tests

  • One mod tests: 116 fns; #[test] 71, #[tokio::test 1, #[rstest 45, #[case 101, #[values 52, #[cfg_attr 2.
  • Fixtures: 6538–6586, 6765–6792, 8199–8510.
  • Private access: fields of Wand/PostingIterator; methods maxscore_search, and_advance_target, next, shallow_next, block_max_score, ….

2. Split

Keep wand.rs as the facade (so the #[path] base directory is unchanged). Also git mv the two #[path] files into the new directory and drop the attributes: inverted/wand_maxscore.rs → inverted/wand/maxscore.rs, inverted/wand_intersection.rs → inverted/wand/intersection.rs (module paths wand::maxscore / wand::intersection are unchanged, so their super::super::builder imports still resolve). Avoid names of inverted siblings (documents, encoding, impact, index, query, scorer, builder).

File Contents (current lines) ~lines
wand.rs (facade) 1–77 (imports, mod lines now without #[path]), 78–237 (TopKCollector, shared by search/maxscore/bulk), 238–333 (tuning), 464–490 (PostingIterator struct), 1823–1991 (MaxScore helpers), 2306–2515 (head/tail, AndWindowStats, Wand struct, atomic_store_max_f32), 6371–6420 (float helpers), private use child::… re-imports for siblings and maxscore.rs, pub(super) use for the items that were pub(super) ~830
wand/posting.rs 334–463 (SIMD, bounds, GroupedTermScorer), 491–1822 (CompressedState, BlockMaxScore, impl PostingIterator, validate_modern_posting_doc_ids) ~1,480
wand/doc_source.rs 1992–2305 (DocCandidate, FlatDocuments, WandDocuments, ModernVisibility, AllModernDocuments, ModernWandDocuments, LegacyWandDocuments, test impl for DocSet) ~330
wand/search.rs impl Wand: 2516–2965 (construction, search, flat_search), 3664–4106 (score-first-AND helpers, next, next_and_candidate), 5144–5639 (OR head/tail) ~1,400
wand/maxscore.rs existing 732 lines + 2966–3663 (maxscore_search) ~1,440
wand/and_bulk.rs 4107–5143 (posting_block_up_to, and_bulk_search, and_move_to_next_block, and_advance_target) ~1,050
wand/phrase.rs 5640–6035 (check_positions, check_exact_positions*, exact_phrase_*, #[cfg(test)] counters, PositionCursor, …) ~410
wand/cursor.rs 6036–6370 (WandCursor, compound_global_score_upper_bound, seek, compound_shallow_bound) ~350
wand/intersection.rs existing file unchanged 132
wand/tests/mod.rs 6421–6427, 6517–6586, 6765–6792, 8199–8510, 8596–8604 (imports + fixtures: CostOnlyDocuments, scorers, generators; the super::super::documents / impact imports stay correct at wand::tests depth) ~430
wand/tests/basics.rs 6428–6516, 6587–6764, 6793–6877, 8017–8198, 8511–8595, 11000–11272 (bounds, norm/topk/env parsing, find_next_geq, position cursor misc, flat/blocks/validation) ~900
wand/tests/maxscore_search.rs 6878–8016 (not named maxscore: tests at 7132/7240 write maxscore::MaxScoreSoaScratch::new(), and a test module named maxscore would shadow the glob-imported wand::maxscore child) ~1,140
wand/tests/or_and.rs 8605–9648 ~1,050
wand/tests/score_first_and.rs 9649–10999 ~1,350
wand/tests/phrase.rs 11273–11972 ~700
wand/tests/bulk_and.rs 11973–12818 ~850

3. Cautions

  • Visibility: items that were pub(super) (visible to inverted) and move into a child must be declared pub(in crate::scalar::inverted) in the child and re-exported from the facade with pub(super) use …; so compound.rs, cross_column.rs and the index.rs glob see the same names as today. Exact facade re-export list: pub use doc_source::DocCandidate; (the only pub item that leaves); pub(super) use posting::{GroupedTermScorer, validate_modern_posting_doc_ids}; pub(super) use doc_source::{FlatDocuments, WandDocuments, ModernVisibility, AllModernDocuments, ModernWandDocuments, LegacyWandDocuments}; pub(super) use cursor::WandCursor;. PostingIterator, Wand, FLAT_SEARCH_PERCENT_THRESHOLD, SharedNormAddends, score_sum_upper_bound_factor, outward_f32_upper_bound stay in the facade. Private items shared between children become pub(super); the facade re-imports them with a private use (never pub(super) use), otherwise new names leak into index.rs's wand::* glob.
  • Methods of Wand/PostingIterator moved to a child and called from another child, from maxscore.rs, or from tests become pub(super) (method visibility is not linted). Fields stay private because both structs stay in the facade. Free functions in children called by tests also need pub(super): find_next_geq_in_block (posting.rs, test 8190) and the #[cfg(test)] phrase counters count_anchor_steps, take_position_cursor_calls, take_anchor_steps, counted_anchor_steps (phrase.rs, tests 11454–11932).
  • Bare child-module paths: and_bulk.rs needs use super::intersection; (intersection::intersect at 4496); inside maxscore.rs the moved line 3003 maxscore::MaxScoreSoaScratch::new() becomes MaxScoreSoaScratch::new() (the type is defined in that same file); the two test sites keep maxscore::… and resolve it through tests/mod.rs's use super::*.
  • The two giant functions move whole with their local macro_rules! and cfg(target_arch) blocks. #[cfg(test)] use std::cell::Cell; (4–5) stays in the facade because the Wand struct's #[cfg(test)] field phrase_position_checks: Cell<usize> (2494) stays there; phrase.rs needs its own #[cfg(test)] use std::cell::Cell; for the thread_local! counters (5863–5908).
  • #[cfg_attr(coverage, coverage(off))] on two generators stays with them in tests/mod.rs.
  • cargo test -p lance-index --lib -- inverted::wand should still list 116 tests; attribute counts: #[test] 71, #[tokio::test 1, #[rstest 45, #[case 101, #[values 52.
  • Delete rust/lance-index/src/scalar/inverted/wand.rs from ci/module_size_exceptions.txt.
  • Verify: cargo test -p lance-index --lib -- scalar::inverted (whole inverted tree, because of the glob), cargo bench -p lance-index --no-run, and the lance crate FTS tests cargo test -p lance --lib -- fts.

4. Verification

Run before and after the move (all from the repo root):

# every new file < 3,000 lines and < 400 KB; CI replay against main
wc -lc <facade.rs> <facade>/*.rs <facade>/tests/*.rs
python3 ci/check_module_size.py main HEAD
# test invariant: attribute counts and the multiset of test fn names are unchanged
grep -hoE '#\[(test|tokio::test|rstest|case|values|serial_test::serial|test_log::test|lance_test_macros::test)' <old file | new files> | sort | uniq -c
cargo test -p <crate> --lib -- --list 2>/dev/null | grep ': test$' | sed 's/.*:://' | sort > names.{before,after}.txt && diff names.before.txt names.after.txt
# exactly what CI runs
cargo clippy --locked --features "$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | sort -u | paste -s -d ,)" --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
cargo fmt --all -- --check
# module-specific commands: see the last bullet of section 3
(cd python && cargo check)          # package pylance
(cd java/lance-jni && cargo check)  # package lance-jni

Pure-move checklist for the PR description: no function body changed, no item renamed, no signature changed, visibility only widened where listed in section 3, every pre-existing path still resolves (re-exports), exceptions-list line removed (if listed), attribute counts and test names identical.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.