Refactor and Split `rust/lance-index/src/scalar/inverted/wand.rs` into submodules
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start with rust/lance-index/src/scalar/inverted/wand.rs, the facade that becomes the parent of the new wand/ directory, and the two #[path] files wand_maxscore.rs and wand_intersection.rs. Before moving anything, run cargo test -p lance-index --lib -- inverted::wand and save the sorted test names, since they must match afterwards. Done means a pure move with no body, name or signature changes, the test names and attribute counts unchanged, the index.rs glob exporting the same names, and the module-size checks passing.
Written by the indexing model from the issue text.
Description
This is a pure move: code is relocated into submodules, no logic changes, 100% path compatibility through re-exports. The mechanics follow PR #9989 (dataset/blob.rs split). Line numbers below refer to the commit above.
1. Current module
Role. WAND / MaxScore / bulk-AND search over posting lists.
Header and existing children
- 4–5
#[cfg(test)] use std::cell::Cell;; imports 30–50. - 25–28
#[path = "wand_intersection.rs"] mod intersection;and#[path = "wand_maxscore.rs"] mod maxscore;(files live ininverted/). wand_maxscore.rs(732 lines) already holdsimpl PostingIterator/impl Wandpieces withpub(super)methods and reads private fields of both structs.
Concern groups
- 52–77 consts/types/norm cache.
- 78–237
TopKCollector. - 238–333 tuning/env:
pub static FLAT_SEARCH_PERCENT_THRESHOLD,BulkAndMode,CompetitiveFloorMode. - 334–403 SIMD
find_next_geq*; 404–426 bounds; 427–463pub(super) GroupedTermScorer. - 464–490
pub struct PostingIterator+CompressedState/BlockMaxScore; 807–1778impl PostingIterator(972 lines); 1779–1822validate_modern_posting_doc_ids. - 1823–1991 MaxScore helpers.
- 1992–2305 document sources:
DocCandidate,FlatDocuments,trait WandDocuments,ModernWandDocuments,LegacyWandDocuments,#[cfg(test)] impl WandDocuments for DocSet. - 2306–2421 head/tail postings.
- 2422–2503
pub struct Wand(many#[cfg(test)]counter fields). - 2516–5725
impl Wand(3,210 lines):- 2520–2702 construction; 2704–2964
search/flat_search - 2966–3662
maxscore_search= one 697-line fn with a localmacro_rules! - 3664–3866 score-first-AND helpers; 3868–4105 classic
next - 4115–5023
and_bulk_search= one 909-line fn with a localmacro_rules!and x86_64cfgblocks - 5144–5639 OR head/tail; 5641–5723 phrase checks
- 2520–2702 construction; 2704–2964
- 5726–6035 phrase scan + position cursors.
- 6036–6370
pub(super) WandCursor+ a secondimpl Wand. - 6371–6420 float helpers.
Visibility and cfg facts
pub(super)items: 11 top-level (429, 1782, 1831, 2005, 2007, 2033, 2046, 2087, 2176, 6041, 6381) and 20 methods.pub(crate)methods at 957, 967, 1063, 1068, 2093, 2102, 2182, 2520, 2638, 2646, 2706.- 76
#[cfg(test)]instrumentation sites in non-test code;target_arch/target_featureat 334–397, 4407–4818.
Wiring
inverted.rs:23 mod wand;(private).
Consumers
compound.rs:33-36,cross_column.rs:34-36.index.rs:72 use super::{…, wand::*}, whose glob flows intoindex/partition.rs,index/search.rs,index/search_candidates.rs,index/posting_reader.rs,index/tests/scoring.rs.
Tests
- One
mod tests: 116 fns;#[test]71,#[tokio::test1,#[rstest45,#[case101,#[values52,#[cfg_attr2. - Fixtures: 6538–6586, 6765–6792, 8199–8510.
- Private access: fields of
Wand/PostingIterator; methodsmaxscore_search,and_advance_target,next,shallow_next,block_max_score, ….
2. Split
Keep wand.rs as the facade (so the #[path] base directory is unchanged). Also git mv the two #[path] files into the new directory and drop the attributes: inverted/wand_maxscore.rs → inverted/wand/maxscore.rs, inverted/wand_intersection.rs → inverted/wand/intersection.rs (module paths wand::maxscore / wand::intersection are unchanged, so their super::super::builder imports still resolve). Avoid names of inverted siblings (documents, encoding, impact, index, query, scorer, builder).
| File | Contents (current lines) | ~lines |
|---|---|---|
wand.rs (facade) |
1–77 (imports, mod lines now without #[path]), 78–237 (TopKCollector, shared by search/maxscore/bulk), 238–333 (tuning), 464–490 (PostingIterator struct), 1823–1991 (MaxScore helpers), 2306–2515 (head/tail, AndWindowStats, Wand struct, atomic_store_max_f32), 6371–6420 (float helpers), private use child::… re-imports for siblings and maxscore.rs, pub(super) use for the items that were pub(super) |
~830 |
wand/posting.rs |
334–463 (SIMD, bounds, GroupedTermScorer), 491–1822 (CompressedState, BlockMaxScore, impl PostingIterator, validate_modern_posting_doc_ids) |
~1,480 |
wand/doc_source.rs |
1992–2305 (DocCandidate, FlatDocuments, WandDocuments, ModernVisibility, AllModernDocuments, ModernWandDocuments, LegacyWandDocuments, test impl for DocSet) |
~330 |
wand/search.rs |
impl Wand: 2516–2965 (construction, search, flat_search), 3664–4106 (score-first-AND helpers, next, next_and_candidate), 5144–5639 (OR head/tail) |
~1,400 |
wand/maxscore.rs |
existing 732 lines + 2966–3663 (maxscore_search) |
~1,440 |
wand/and_bulk.rs |
4107–5143 (posting_block_up_to, and_bulk_search, and_move_to_next_block, and_advance_target) |
~1,050 |
wand/phrase.rs |
5640–6035 (check_positions, check_exact_positions*, exact_phrase_*, #[cfg(test)] counters, PositionCursor, …) |
~410 |
wand/cursor.rs |
6036–6370 (WandCursor, compound_global_score_upper_bound, seek, compound_shallow_bound) |
~350 |
wand/intersection.rs |
existing file unchanged | 132 |
wand/tests/mod.rs |
6421–6427, 6517–6586, 6765–6792, 8199–8510, 8596–8604 (imports + fixtures: CostOnlyDocuments, scorers, generators; the super::super::documents / impact imports stay correct at wand::tests depth) |
~430 |
wand/tests/basics.rs |
6428–6516, 6587–6764, 6793–6877, 8017–8198, 8511–8595, 11000–11272 (bounds, norm/topk/env parsing, find_next_geq, position cursor misc, flat/blocks/validation) |
~900 |
wand/tests/maxscore_search.rs |
6878–8016 (not named maxscore: tests at 7132/7240 write maxscore::MaxScoreSoaScratch::new(), and a test module named maxscore would shadow the glob-imported wand::maxscore child) |
~1,140 |
wand/tests/or_and.rs |
8605–9648 | ~1,050 |
wand/tests/score_first_and.rs |
9649–10999 | ~1,350 |
wand/tests/phrase.rs |
11273–11972 | ~700 |
wand/tests/bulk_and.rs |
11973–12818 | ~850 |
3. Cautions
- Visibility: items that were
pub(super)(visible toinverted) and move into a child must be declaredpub(in crate::scalar::inverted)in the child and re-exported from the facade withpub(super) use …;socompound.rs,cross_column.rsand theindex.rsglob see the same names as today. Exact facade re-export list:pub use doc_source::DocCandidate;(the onlypubitem that leaves);pub(super) use posting::{GroupedTermScorer, validate_modern_posting_doc_ids}; pub(super) use doc_source::{FlatDocuments, WandDocuments, ModernVisibility, AllModernDocuments, ModernWandDocuments, LegacyWandDocuments}; pub(super) use cursor::WandCursor;.PostingIterator,Wand,FLAT_SEARCH_PERCENT_THRESHOLD,SharedNormAddends,score_sum_upper_bound_factor,outward_f32_upper_boundstay in the facade. Private items shared between children becomepub(super); the facade re-imports them with a privateuse(neverpub(super) use), otherwise new names leak intoindex.rs'swand::*glob. - Methods of
Wand/PostingIteratormoved to a child and called from another child, frommaxscore.rs, or from tests becomepub(super)(method visibility is not linted). Fields stay private because both structs stay in the facade. Free functions in children called by tests also needpub(super):find_next_geq_in_block(posting.rs, test 8190) and the#[cfg(test)]phrase counterscount_anchor_steps,take_position_cursor_calls,take_anchor_steps,counted_anchor_steps(phrase.rs, tests 11454–11932). - Bare child-module paths:
and_bulk.rsneedsuse super::intersection;(intersection::intersectat 4496); insidemaxscore.rsthe moved line 3003maxscore::MaxScoreSoaScratch::new()becomesMaxScoreSoaScratch::new()(the type is defined in that same file); the two test sites keepmaxscore::…and resolve it throughtests/mod.rs'suse super::*. - The two giant functions move whole with their local
macro_rules!andcfg(target_arch)blocks.#[cfg(test)] use std::cell::Cell;(4–5) stays in the facade because theWandstruct's#[cfg(test)]fieldphrase_position_checks: Cell<usize>(2494) stays there;phrase.rsneeds its own#[cfg(test)] use std::cell::Cell;for thethread_local!counters (5863–5908). #[cfg_attr(coverage, coverage(off))]on two generators stays with them intests/mod.rs.cargo test -p lance-index --lib -- inverted::wandshould still list 116 tests; attribute counts:#[test]71,#[tokio::test1,#[rstest45,#[case101,#[values52.- Delete
rust/lance-index/src/scalar/inverted/wand.rsfromci/module_size_exceptions.txt. - Verify:
cargo test -p lance-index --lib -- scalar::inverted(whole inverted tree, because of the glob),cargo bench -p lance-index --no-run, and thelancecrate FTS testscargo test -p lance --lib -- fts.
4. Verification
Run before and after the move (all from the repo root):
# every new file < 3,000 lines and < 400 KB; CI replay against main
wc -lc <facade.rs> <facade>/*.rs <facade>/tests/*.rs
python3 ci/check_module_size.py main HEAD
# test invariant: attribute counts and the multiset of test fn names are unchanged
grep -hoE '#\[(test|tokio::test|rstest|case|values|serial_test::serial|test_log::test|lance_test_macros::test)' <old file | new files> | sort | uniq -c
cargo test -p <crate> --lib -- --list 2>/dev/null | grep ': test$' | sed 's/.*:://' | sort > names.{before,after}.txt && diff names.before.txt names.after.txt
# exactly what CI runs
cargo clippy --locked --features "$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | sort -u | paste -s -d ,)" --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
cargo fmt --all -- --check
# module-specific commands: see the last bullet of section 3
(cd python && cargo check) # package pylance
(cd java/lance-jni && cargo check) # package lance-jni
Pure-move checklist for the PR description: no function body changed, no item renamed, no signature changed, visibility only widened where listed in section 3, every pre-existing path still resolves (re-exports), exceptions-list line removed (if listed), attribute counts and test names identical.
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 900
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 366
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 78/100
lance-format/lance#10225 ·
Maintainers usually reply within 3 days
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 3 days
-
After initialize_mem_wal, every FTS query logs "ERROR Index __lance_mem_wal has no fields"Possibly taken @lance-gatefixer claimed this 24 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
lance-format/lance#9655 ·
Maintainers usually reply within 3 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 3 days
All issues in lance-format/lance
Similar issues
-
Small glitchOpenpriority: medium
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
storytold/photocraft#9538 ·
Maintainers usually reply within 1 day
-
bug P2 rust
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
jscott3201/rusty-bacnet#2085 ·
Maintainers usually reply within 1 day
-
area:tag area:tag---json---markdown--n bug factory:dogfood factory:triaged persona:scripter platform:linux-cloud priority:p3 type:bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
gitbutlerapp/grit#1940 · 1 comment ·
Maintainers usually reply within 1 day
-
Inconsistent case in names for Windows installation packages compared to all other assetsPossibly taken A pull request linked to this issue is open or already merged. Openarea:ui bug in progress needs-triage
Difficulty 1/5 Under an hour Newbie friendliness 65/100
bherbruck/solvecraft#6769 · 2 comments ·