Expose score explanations for relevance debugging

Start with the public API in ts/native.ts, the native implementation in src/engine.rs, and the README wrapper contract; compare Tantivy’s Explanation capability with the existing search path. Define a bounded, typed explanation interface tied to an explicit reader snapshot, then add coverage for the listed query types and limits. Done means the acceptance criteria are met and the diagnostic-only behavior is documented.

Outcome

Expose privileged, snapshot-bound explanations for one to ten IDs. Tantivy owns indexed execution,
scorers and explanation arithmetic. Fulltext owns typed compilation, admission, lifecycle, ID
projection and response validation. Harper owns authorization and current-source policy.

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API.
Implementation: Expose snapshot-bound score explanations (blocked draft).

Current evidence

The root repair is Preserve scorer boost context in explanations,
with Do not seek backwards when explaining a document as
its prerequisite. Fulltext reconstructs no scores and sanitizes no native trees.

Local performance qualification retains 96 raw rows covering distributed one/ten IDs, six query
shapes, up to 131,072 documents/24 observed segments, default/raised policy and a separate raised-policy
1,024-field smoke. Every row passes scorer parity and observed-sample tripwires. The wide-schema
smoke uses a 128 MB writer; default, stress and ordinary raised profiles use 15 MB. The wide-schema 15 MB setup was interrupted before timing and is not qualified.
Native per-tree work remains uninterruptible; these observations establish no strict deadline,
per-request allocation bound or universal catalog capacity.

The completion-aware Node runner continues genuine debug assertion failures into production,
keeps production as the last native build, and aborts setup/interruption/abnormal outcomes.
Its 14 real-subprocess fixtures pass locally on Node 22.18.0, 24.16.0 and 26.10.0, without skips (the corresponding node-completion-<version>.log files). Rust 1.90.0 is recorded in explain-027-rust-toolchain.log; explain-027-native-eight-debug.log and explain-027-native-eight-release.log record the 8/8 native runs.

Release acceptance

• Apply the diagnostic 256-clause ceiling during construction through the shared compiler, preserve lower limits, and preflight facade structure before queue admission.
• Retain source-qualified raw cost evidence and wire default/raised smoke gates into existing Linux CI jobs; keep larger stress profiles manual. Hosted green production evidence against the adopted native release remains pending.
• Exercise both Node phases after completed debug assertion failures and distinguish interrupted/unattested processes from assertion verdicts.
• Adopt an approved native release containing the seek/context fixes and pass all typed-query regressions against that committed dependency. Tantivy current main requires Rust 1.91 while fulltext pins 1.90; require a compatible backport or an approved toolchain update.
• Verify hosted platform CI at the release head, including production qualification.
• Assess native interruptibility and unqualified workload distributions before promising operational limits beyond the measured diagnostic envelope.
• Select an unused synchronized facade/Cargo/platform version and matching prebuilds. Published 0.9.0 is ABI 14; this draft is ABI 15/query API 8 and must reconcile with Boolean tracing's ABI 16/query API 9.
• Preserve privilege checks, exact scorer bits, stale/retained snapshots, filters/candidates, and missing/nonmatching ID behavior in Harper integration tests.

Harper adoption is tracked in Expose authorized snapshot-bound full-text score explanations.
No Harper runtime adoption or package publication is claimed. The existing search mode/operator
validation-order cleanup remains a separate deferred item.

Review follow-up

The added searchPage().snapshot property is intentional, documented diagnostic metadata, including on the final page. It is not a runtime defect or an authorization token. Keep this contract in the API documentation and consumer coverage.

Review-Deferral: 1561721ee4810780 severity=none category=documentation

The native follow-up implements phrase-prefix BM25 details using the scorer’s leading-phrase occurrence count rather than the suffix-qualified count, without changing scorer arithmetic. Fulltext’s current prefix compiler uses term/fuzzy expansion, so that native phrase-prefix path does not affect its compiler. Native follow-up commit 66741044 passes the new frequency regression in debug and release, the existing 8-test explanation target in both profiles, 1,345 default and 1,401 all-feature library tests (8 ignored each), and 54 doctests (2 ignored), plus Clippy and formatting. The final test/documentation follow-up b3924c17 adds explicit repeated-frequency assertions for phrase and regex-phrase as well, with the dedicated target again passing 8/8 in debug and release. Earlier published-0.27 backport and 96-row cost evidence above qualified the feda025 implementation before this follow-up; they do not qualify a newly released artifact. The approved native dependency, Rust 1.91 versus fulltext 1.90 adoption decision, synchronized artifacts and hosted release gates above remain open.

Comment generated by kAIle (GPT-6)