Refactor and Split `rust/lance/src/dataset/optimize.rs` into submodules

Start with rust/lance/src/dataset/optimize.rs and the line ranges in the table, moving code into the optimize/ submodules while keeping the //! doctest and imports in the facade. The visibility changes in section 3 and the re-exports must keep every existing path resolving. Done means cargo test -p lance --lib -- dataset::optimize, cargo test -p lance --doc, and the test-name diff are all clean, with no function bodies changed.

This is a pure move: code is relocated into submodules, no logic changes, 100% path compatibility through re-exports. The mechanics follow PR #9989 (dataset/blob.rs split). Line numbers below refer to the commit above.

1. Current module

Role. Compaction.

Header and existing children

• 4–82 //! docs with a compiled doctest using lance::dataset::optimize::compact_files and IgnoreRemap (must stay in the facade).
• 84–149 imports (use super::{fragment::FileFragment, …, cleanup_data_fragments, write::write_fragments_internal_with_file_row_counts}).
• 150 pub(super) mod binary_copy; (used by dataset/versions/mod.rs:354); 151 pub mod remapping; (4,989 lines, out of scope); 155 pub use remapping::{IgnoreRemap, IndexRemapper, IndexRemapperOptions, RemappedIndex};.

Concern groups

• 157–610 options: CompactionMode, IndexRemapMode, CompactionOptions + #[allow(deprecated)] impls, COMPACTION_CONFIG_PREFIX.
• 612–755 binary-copy eligibility: can_use_binary_copy, first_binary_copy_version_mismatch, pub(super) can_use_binary_copy_current (used by versions/mod.rs:334).
• 757–778 CompactionMetrics.
• 780–1320, 2142–2406 planning: CompactionPlanner, DefaultCompactionPlanner, compact_files, compact_files_with_planner, FragmentMetrics, CompactionPlan, TaskData, CompactionTask, CandidateBin, load_index_fragmaps, reject_unremappable_rewrite, plan_compaction.
• 1322–2066 blob v2 rewrite: BlobV2FieldRewritePlan, pub(crate) BlobV2BatchRewritePlan, pub(crate) field_contains_blob_v2, transform_blob_v2_batch; inline super::blob:: at 1493, 1814.
• 2068–2140 prepare_reader; 2408–2883 rewrite execution: RewriteResult, reserve_fragment_ids, live_row_addrs, rewrite_files (362 lines).
• 2885–3269 row lineage (PlannedRowLineage with private fields built by rewrite_files at 2793).
• 3271–3809 commit: commit_compaction (489 lines), normalize_source_fragments, cleanup helper.

Wiring

• dataset.rs:82 pub mod optimize;

Consumers

• compact_files/CompactionOptions: ~25 files incl. java blocking_dataset.rs/utils.rs.
• CompactionMetrics, CompactionMode, CompactionPlan, CompactionTask, RewriteResult, commit_compaction, plan_compaction, TaskData, IndexRemapperOptions: python/src/dataset/optimize.rs and java/lance-jni/src/optimize.rs.
• BlobV2BatchRewritePlan: blob/clone.rs, write/update.rs, merge_insert/exec/write.rs, fragment.rs:2075. field_contains_blob_v2: write/update.rs.

Tests

• mod tests { mod binary_copy; mod concurrent_reindex; … } (3812–3815, backed by optimize/tests/{binary_copy,concurrent_reindex}.rs).
• 133 fns; #[test] 13, #[tokio::test 117, #[rstest 35, #[case 101, #[values 25; three rstest-only fns.
• Private access: 11 tests call rewrite_files; others use can_use_binary_copy, live_row_addrs, reserve_fragment_ids, lineage fns, CandidateBin, BlobV2FieldRewritePlan, normalize_source_fragments. optimize/tests/binary_copy.rs uses can_use_binary_copy; optimize/remapping.rs tests import rewrite_files.

2. Split

Keep optimize.rs as the facade (directory exists). Avoid binary_copy, remapping, tests.

File	Contents (current lines)	~lines
optimize.rs (facade)	1–156 (docs + doctest, imports, mod lines, pub use remapping::…), 756–779 (CompactionMetrics), 1028–1105 (compact_files, compact_files_with_planner), 2399–2407 (plan_compaction), new mod lines, re-exports, private use re-imports of items optimize/tests/* reach through super::*, #[cfg(test)] mod tests;	~300
optimize/options.rs	157–611 + unit tests 11925–12185 (from_dataset_config ×11)	~720
optimize/plan.rs	780–1027, 1106–1321, 2141–2398 + unit test 3876–3959 (test_candidate_bin)	~810
optimize/blob_v2.rs	1322–2067 + unit test 12959–13009	~800
optimize/rewrite.rs	612–755 (binary-copy eligibility), 2068–2140 (prepare_reader), 2408–2884 (RewriteResult, reserve_fragment_ids, live_row_addrs, rewrite_files)	~700
optimize/lineage.rs	2885–3270 + unit tests 5137–5450	~710
optimize/commit.rs	3271–3810 + unit test 14494–14530	~580
optimize/tests/mod.rs	3812–3875 (header, mod binary_copy; mod concurrent_reindex;, imports), shared helpers 3960–3983, 4408–4432, 4818–4854, 6310–6368, 8648–8686; topic mod lines	~260
optimize/tests/compaction.rs	3984–4072 (mock remapper), 4073–5136, 5451–5858 (incl. the test-local IgnoreRemap 5644–5659), 11882–11924, 12186–12477, 12478–12882, 14075–14493	~2,640
optimize/tests/stable_row_ids.rs	5859–6767, 9093–10063	~1,820
optimize/tests/defer_remap.rs	6768–8647, 8687–9092	~2,290
optimize/tests/index_remap.rs	10064–11881	~1,820
optimize/tests/blob_v2.rs	12883–14074 minus 12959–13009	~1,160
existing optimize/tests/binary_copy.rs, optimize/tests/concurrent_reindex.rs, optimize/binary_copy.rs, optimize/remapping.rs	unchanged	—

Facade re-exports: pub use options::{CompactionMode, IndexRemapMode, CompactionOptions, COMPACTION_CONFIG_PREFIX}; pub use plan::{CompactionPlanner, DefaultCompactionPlanner, CompactionPlan, TaskData, CompactionTask}; pub use rewrite::RewriteResult; pub use commit::commit_compaction; pub(crate) use blob_v2::{BlobV2BatchRewritePlan, field_contains_blob_v2, transform_blob_v2_batch}; pub(in crate::dataset) use rewrite::can_use_binary_copy_current; plus private use rewrite::{rewrite_files, can_use_binary_copy, …}; for the test subtree and remapping.rs tests (which import crate::dataset::optimize::rewrite_files).

3. Cautions

• can_use_binary_copy_current was pub(super): declare pub(in crate::dataset) in rewrite.rs and re-export. pub(crate) blob items become pub in blob_v2.rs + pub(crate) use in the facade.
• pub(super) needed: rewrite_files (also for remapping.rs tests via the facade re-import), reserve_fragment_ids, live_row_addrs, can_use_binary_copy, first_binary_copy_version_mismatch, load_index_fragmaps, reject_unremappable_rewrite, compute_row_lineage, rechunk_row_lineage, place_planned_row_lineage, append_row_lineage_columns, records_stable_row_id_moves, pins_dead_user_columns_for_lineage, PlannedRowLineage + its fields (literal in rewrite_files), CompactionPlan::{new, extend_tasks, tasks}, CompactionOptions::write_version if called outside options.rs.
• The doctest in //! (4–82) stays in the facade; cargo test -p lance --doc dataset::optimize must pass.
• Inline super::blob:: at 1493/1814 (blob_v2.rs) → crate::dataset::blob::; commit.rs needs use super::remapping; for the bare remapping::transpose_row_addrs at 3447; test 3816 use self::remapping::RemappedIndex; works from tests/mod.rs through use super::*.
• #[allow(deprecated)] on the two CompactionOptions impls move with them.
• python/python/lance/optimize.py:6 mentions src/dataset/optimize.rs in a comment; leave it.
• Test invariant: 133 fns (incl. 3 rstest-only); #[test] 13, #[tokio::test 117, #[rstest 35, #[case 101, #[values 25.
• Delete rust/lance/src/dataset/optimize.rs from ci/module_size_exceptions.txt.
• Verify: cargo test -p lance --lib -- dataset::optimize; cargo test -p lance --doc; cargo check in python/ and java/lance-jni/ (both import the optimize types).

4. Verification

Run before and after the move (all from the repo root):

# every new file < 3,000 lines and < 400 KB; CI replay against main
wc -lc <facade.rs> <facade>/*.rs <facade>/tests/*.rs
python3 ci/check_module_size.py main HEAD
# test invariant: attribute counts and the multiset of test fn names are unchanged
grep -hoE '#\[(test|tokio::test|rstest|case|values|serial_test::serial|test_log::test|lance_test_macros::test)' <old file | new files> | sort | uniq -c
cargo test -p <crate> --lib -- --list 2>/dev/null | grep ': test$' | sed 's/.*:://' | sort > names.{before,after}.txt && diff names.before.txt names.after.txt
# exactly what CI runs
cargo clippy --locked --features "$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | sort -u | paste -s -d ,)" --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
cargo fmt --all -- --check
# module-specific commands: see the last bullet of section 3
(cd python && cargo check)          # package pylance
(cd java/lance-jni && cargo check)  # package lance-jni

Pure-move checklist for the PR description: no function body changed, no item renamed, no signature changed, visibility only widened where listed in section 3, every pre-existing path still resolves (re-exports), exceptions-list line removed (if listed), attribute counts and test names identical.