Refactor and Split `rust/lance/src/dataset/mem_wal/write.rs` into submodules

Start with rust/lance/src/dataset/mem_wal/write.rs and the line ranges in the issue, then check write/tests/evolve_schema.rs for the super::super::* imports it depends on. Move the code into the listed submodules, widen visibility only where the compiler requires it, and delete the entry from ci/module_size_exceptions.txt. Done when cargo test -p lance --lib -- dataset::mem_wal::write passes and the test name list is unchanged.

This is a pure move: code is relocated into submodules, no logic changes, 100% path compatibility through re-exports. The mechanics follow PR #9989 (dataset/blob.rs split) . Line numbers below refer to the commit above.

1. Current module

Role. The MemWAL shard writer.

Header

• 4 #![allow(clippy::print_stderr)]; 6–13 //! docs with intra-doc links; 15–64 imports.
• 40–47 pub use super::{…} re-exports: BTreeMemIndex, IndexStore, MemIndexSpec, validate_index_specs, CacheConfig, MemTable, BatchStore, StoreFull, StoredBatch, MemTableFlusher, MemTableScanner, WatchableOnceCell(Reader), WalEntry, WalEntryData, WalFlushFailure, WalFlushResult, WalFlusher.

Concern groups

• 66–461 ShardWriterConfig.
• 462–696 task executor: MessageHandler, TaskDispatcher, TaskExecutor.
• 697–1278 durability + backpressure + ShardMemory.
• 1279–1427 shared writer state types: WriteResult, FrozenMemTable, ResidentMemTables, publish_memory, WriterState, in_memory_ref, time helpers.
• 1428–1704, 1819–1860 WAL replay.
• 1705–1818 seal policy.
• 1861–2197 schema conformance: WriterSchema, Incoming, tombstones.
• 2198–2718 SharedWriterState + WAL triggers.
• 2719–4408 ShardWriter (1,690 lines): open 2754–3297, put/delete 3299–3755, accessors 3757–3953, evolve/seal 3955–4185, shutdown 4187–4406.
• 4409–4512 stats/fence.
• 4513–5057 background handlers.
• 5058–5330 WriteStats.

Wiring

• mem_wal.rs:48 pub mod write;
• mem_wal.rs:331-334 pub use write::{SealFence, ShardWriter, ShardWriterConfig, WriteResult};

Consumers (via mem_wal::write::)

• BatchStore/IndexStore from ~20 scanner files; MemIndexSpec.
• DurabilityResult/WalFlushResult: memtable.rs, wal.rs.
• MemTableStats/ShardMemory/WriteStatsSnapshot: python/src/mem_wal.rs:24, java/lance-jni/src/mem_wal.rs:32, benches.

Tests

• mod tests (5331–12219) with mod evolve_schema; (12218) loading the existing write/tests/evolve_schema.rs (uses super::super::* and write::shard_writer_tests::{create_test_schema, create_test_batch}).
• #[cfg(test)] mod shard_writer_tests (12221–13686): pub(super) fixtures, one #[ignore].
• Private fields accessed: mode, wal_flusher, epoch, task_executor, memtable, schema, frozen_memtables.
• Private fns accessed: conform_to_storage_schema, build_tombstone_batch, claim_*, next_pending_store, memtable_reached_flush_threshold, STALL_GRACE, resolve_backpressure, in_memory_ref, same_index_set, merge_close_stage, LocalBackpressureController::new, TaskExecutor::new, WriterMode.

2. Split

Keep write.rs as the facade (write/tests/ exists).

File	Contents (current lines)	~lines
write.rs (facade)	1–65 (allow, docs, imports, pub use super::…), 1279–1427 (shared state types; private fields read by several children), new mod lines, pub use of every moved pub item, #[cfg(test)] mod tests;, #[cfg(test)] mod shard_writer_tests;	~230
write/config.rs	66–461	~400
write/tasks.rs	462–696 (task executor) + 4491–5057 (next_pending_store, IndexApplyHandler, WalFlushHandler, MemTableFlushHandler, sweep_expired_frozen, flush_memtable)	~810
write/backpressure.rs	697–1278	~590
write/replay.rs	1428–1704, 1819–1860	~330
write/schema.rs	1861–2197	~340
write/shared_state.rs	1705–1818 (seal policy), 2198–2718	~640
write/shard_writer.rs	2719–4490 (WriterMode, ShardWriter, MemTableStats, WalStats, SealFence)	~1,780
write/stats.rs	5058–5330	~280
write/tests/mod.rs	5331–5361, 6124–6161, 6893–6980, 7334–7355, 9070–9082, 9405–9437, 11972–12016 (shared fixtures), mod evolve_schema;, topic mod lines; must keep every import name evolve_schema.rs reaches through super::super::*	~280
write/tests/blob.rs	5362–6101	~740
write/tests/unit.rs	6102–6123, 6162–6543, 7556–7570, 9019–9069, 10109–10223	~590
write/tests/delete.rs	6544–6892, 6981–7333	~700
write/tests/write_flush.rs	7356–7555, 7571–8094, 8259–8499	~970
write/tests/backpressure.rs	8095–8258, 8500–9018	~690
write/tests/wal_only.rs	9083–9404, 10813–10989	~500
write/tests/replay.rs	9438–10108, 10224–10812	~1,260
write/tests/seal.rs	10990–11971, 12017–12217	~1,190
write/shard_writer_tests.rs	12220–13686 unchanged (module name is referenced by evolve_schema.rs and mirrored in python/python/tests/test_mem_wal.py:351)	~1,470

Facade re-exports: pub use config::ShardWriterConfig; pub use tasks::{MessageHandler, TaskExecutor}; pub use backpressure::{DurabilityResult, DurabilityWatcher, DurabilityCell, BackpressureStats, BackpressureWaitGuard, BackpressureStatsSnapshot, ShardMemory, Drain, BackpressureController, LocalBackpressureController}; pub use shard_writer::{ShardWriter, MemTableStats, WalStats, SealFence}; pub use stats::{WriteStats, WriteStatsSnapshot, SharedWriteStats, new_shared_stats}; (every item that is pub today keeps its mem_wal::write::X path). WriteResult stays in the facade.

3. Cautions

• #![allow(clippy::print_stderr)] stays on the facade (inner attribute applies to descendants).
• 22 super:: imports of mem_wal siblings (index, memtable, wal, util, observer, scanner, reconcile, manifest) in moved code become crate::dataset::mem_wal::….
• Private items called across children become pub(super): resolve_backpressure, publish_memory, in_memory_ref, now_millis, replay_memtable_from_wal, SealOutcome, seal_blocked_error, memtable_reached_flush_threshold, memtable_cannot_take, fill_reached_flush_threshold, flush_replayed_memtable, index_replayed_batches, same_index_set, pk_index_columns, validate_against_logical_schema, conform_live_batch, conform_to_storage_schema, build_tombstone_batch, claim_wal_*, next_pending_store, sweep_expired_frozen, DRAIN_POLL_INTERVAL, STALL_GRACE, WriterSchema, Incoming, SharedWriterState, WalOnlyTriggerState, the three handlers, TaskDispatcher, MessageFactory, WriterMode, merge_close_stage, memtable_state_lock, LocalBackpressureController::new, TaskExecutor::new (the compiler enumerates the rest). Struct fields of WriterState, ResidentMemTables, FrozenMemTable stay private because the structs stay in the facade. ShardWriter moves to shard_writer.rs, and write/tests/*.rs plus shard_writer_tests.rs are not its descendants, so the fields they read become pub(super): mode (5513–6049), wal_flusher (5565–5830), task_executor (8472), epoch, and any other the compiler reports (fields 2745–2750).
• #[cfg(test)] arms inside ShardMemorySource (848, 892–1047) travel with backpressure.rs.
• The //! doc links (ShardWriter, MemTable, WalFlusher, IndexStore, MemTableFlusher) must still resolve from the facade (they do, via the re-exports and pub use super::…).
• Test invariant: 141 fns in write.rs (#[test] 21, #[tokio::test 120, #[rstest 5, #[case 17, #[ignore 1) + 10 in evolve_schema.rs unchanged. Doc comments inside tests mention shard_writer_tests::…; keep the name.
• Delete rust/lance/src/dataset/mem_wal/write.rs from ci/module_size_exceptions.txt.
• Verify: cargo test -p lance --lib -- dataset::mem_wal::write; cargo check in python/ and java/lance-jni/; cargo bench -p lance --no-run (benches import mem_wal::write::…).

4. Verification

Run before and after the move (all from the repo root):

# every new file < 3,000 lines and < 400 KB; CI replay against main
wc -lc <facade.rs> <facade>/*.rs <facade>/tests/*.rs
python3 ci/check_module_size.py main HEAD
# test invariant: attribute counts and the multiset of test fn names are unchanged
grep -hoE '#\[(test|tokio::test|rstest|case|values|serial_test::serial|test_log::test|lance_test_macros::test)' <old file | new files> | sort | uniq -c
cargo test -p <crate> --lib -- --list 2>/dev/null | grep ': test$' | sed 's/.*:://' | sort > names.{before,after}.txt && diff names.before.txt names.after.txt
# exactly what CI runs
cargo clippy --locked --features "$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | sort -u | paste -s -d ,)" --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
cargo fmt --all -- --check
# module-specific commands: see the last bullet of section 3
(cd python && cargo check)          # package pylance
(cd java/lance-jni && cargo check)  # package lance-jni

Pure-move checklist for the PR description: no function body changed, no item renamed, no signature changed, visibility only widened where listed in section 3, every pre-existing path still resolves (re-exports), exceptions-list line removed (if listed), attribute counts and test names identical.