Qualify Tantivy 0.27 upgrade and enforce the analyzer rebuild boundary

Outcome

Upgrade Tantivy to 0.27.0 with an explicit persisted-index rebuild boundary, retaining the public analyzer configuration and bounded search/lifecycle contracts.

Verified evidence

• Bump tantivy from 0.26.2 to 0.27.0 fails compilation on dictionary callbacks and fast-field configuration; its supply-chain check rejects new random-backend duplicate versions.
• A local comparison of rust-stemmers 1.2.0 and frostem 1.20260821.8 English algorithms found 484 changed outputs across 235,976 dictionary entries. Examples: added: ad -> add; acarologist: acarologist -> acarolog. This is local algorithm evidence, not a production incident or exhaustive language proof.
• An unguarded port can open a real 0.26.2-written index, so engine-format detection alone does not enforce analyzer compatibility. The artifact probe also observed small score differences between engines.
• Existing sidecar identities 4/5 encode no engine-analysis revision. Existing reset parsing recognizes earlier identities separately from serving them.

Upgrade acceptance

Completed and platform-qualified at the original PR head before the hydration follow-up.

• Port dictionary and fast-field APIs, update runtime reporting and preserve exact-version supply-chain policy.
• Version persisted identity for the changed analyzer output; inspection and opening reject older indexes before serving or mutation, while reset can recognize/retire old directories.
• Keep english@2 as the public configuration profile; document that the sidecar revision determines persisted compatibility and this upgrade requires rebuilding derived indexes from authoritative source records.
• Preserve scorer/ordinal invariants and ordinary/traversal consistency, with tests established on 0.26.2 first.
• Exercise actual old/new addon artifacts, reset/rebuild and downgrade behavior; do not promise in-place rollback without evidence.
• Run full wrapper/platform/package gates and compare existing release benchmarks before promotion. Keep feature additions separate.

Related: Support versioned per-field analyzers and additional languages covers future selectable analyzers; this task only versions the engine upgrade's persisted identity. No existing open epic specifically owns dependency migration. No customer-specific or security incident is claimed; the standalone release target is v0.7.0; consumer integration remains separate.

Hydration performance follow-up

Implemented in Upgrade fulltext to Tantivy 0.27 with safe index rebuilds: retain document addresses through ordinary pagination, decode versions only for returned hits, and share that final hydration across all four paths. Mandatory-ID checks, ranking, collector routing and the explicit fuzzy constant-score contract remain intact. Skipped version metadata is no longer proactively checked, matching the other search paths.

• Rust and public Node regressions cover offset/tie paths, shared/missing/long versions, exact/lower-bound totals and exhausted pages.
• Full format/lint/Rust/Node/packed-package, optimized concurrency and cargo-deny checks pass locally.
• Three interleaved optimized macOS arm64 probes over 10,000 documents: median-of-run medians 6.570 → 1.182 ms for offset 9,990 / ten returned hits / approximately 2 KiB versions; decoded versions fall from 10,000 to ten by code trace. No-version control 0.797 → 0.790 ms. Warmed shallow-page control (three 1,000-query runs) 0.142 → 0.145 ms. Returned IDs/scores and expected versions match. These synthetic local results do not establish a universal speedup.
• Independent Claude/Composer review completed; final-head platform CI passes.

Remaining work is tracked in Make fuzzy edit distance and transpositions configurable, Reduce repeated matching work during deep ranked traversal, Qualify Tantivy pruning for small-window ordinary ANY searches, and Measure and reduce prefix-query analyzer setup overhead.

Targeted Linux qualification gate

Review-Deferral: aa53b95ed8d66982 severity=minor category=performance

• Run the production-equivalent head's Release benchmark on Linux x64 and arm64, including the retained hydration driver with both exact and approximate totals. Confirm all eight case/profile records per architecture and compare generic query/multi-index metrics against the v0.6.0 baseline, acknowledging shared-runner noise. The new targeted matrix has no historical Linux release baseline; do not claim an isolated Linux speedup from it.
• Keep the PR draft until fresh platform CI, the targeted benchmark jobs and all feedback settle. Update the guarded PR-body evidence and run Ready gates before promotion.

The existing benchmark workflow now invokes the targeted driver in both Linux jobs and retains its JSONL output. This closes the prior coverage gap; remote execution and artifact checks are complete. No merge, package publication or consumer pin change is authorized.

Final qualification evidence

Qualified 7592c53698b66bae572367f17bbca0a908355c7f in Upgrade fulltext to Tantivy 0.27 with safe index rebuilds. Final CI passes Linux x64/arm64, macOS, Windows, supply-chain, optimized concurrency, coverage and smoke gates. Claude and Composer reviewed production-equivalent 70181b9; final changes only retain eventual release JSONL assets and close unexpected test handles. Full local checks pass after those narrow review fixes.

Both upgrade/baseline and upgrade repeat/baseline repeat passed. All eight hydration profiles per architecture were verified in both upgrade runs. Repeat mixed warm p99: x64 8.555 → 7.734 ms; arm64 7.238 → 5.784 ms. The broad first-pair x64 slowdown did not reproduce. Individual short query-mode and cold phases vary; baseline x64 indexing itself ranged 61,235 → 49,470 documents/s across runs. Separate shared runners and Node patch versions limit attribution; this is qualification evidence, not a guaranteed speedup. No historical Linux hydration baseline exists.

The platform deferral above is satisfied. Window-wide ID decoding remains tracked in Qualify Tantivy pruning for small-window ordinary ANY searches. No merge, package publication or Harper pin change was performed.

Additional safe reuse qualification

Upgrade fulltext to Tantivy 0.27 with safe index rebuilds is back in draft for authorized fuzzy-automaton reuse and review cleanups. Prior green qualification applies to7592c53; the new implementation requires fresh checks and trace performance evidence.

Review-Deferral: be7fdeca1e38283d severity=nit category=test-coverage

• Consider retained optimized hydration/trace performance comparisons on Windows and macOS alongside Linux. Platform correctness CI already covers these platforms, and local macOS hydration measurements exist; this follow-up concerns retained cross-platform performance evidence. Do not add CI jobs without measured value or call missing benchmark coverage a proven regression.

Exact-total collection is tracked in Qualify Tantivy pruning for small-window ordinary ANY searches, including native combined collectors versus separate pruned searches.

Current safe reuse head

Current qualification targets 9d6ec6f3da8f190f6c087f9da370f728dd395605 (see the linked PR for the full SHA), superseding prior-head completion evidence. Full local checks, optimized concurrency and dependency-policy checks pass. Fresh Linux upgrade run 40546860048 and baseline 40546866756, plus CI, are pending. Require eight hydration and eight trace cases per architecture. Native reuse is limited to fuzzy-prefix; the measured full-fuzzy preparation regression is deferred to Make fuzzy edit distance and transpositions configurable. Exact-total strategy remains tracked in Qualify Tantivy pruning for small-window ordinary ANY searches.

Final safe-reuse qualification complete

Qualification for 9d6ec6f3da8f190f6c087f9da370f728dd395605 supersedes the pending statements above. Final-head CI is green on Linux x64/arm64, macOS and Windows. Local full checks, optimized concurrency and dependency policy pass. All three paired Release comparisons succeeded; eight distinct hydration and eight trace records per Linux architecture pass, including exact/approximate totals and page assertions.

The longer upgrade/v0.6.0 baseline uses 10,000 mixed queries. With identical Node 24.21.0 on x64, multi-index throughput was 13,980→15,085 queries/s (+7.9%), p99 1.281→1.163ms; ordinary mixed throughput +6.6%, p99 5.576→5.170ms. The earlier short-phase x64 throughput loss did not persist. ARM ordinary mixed throughput +13.1%, multi-index +3.4%; ARM Node versions differ. Individual tails remain variable: x64 range-filter p99 increased 4.651→5.921ms in the long pair but decreased 5.824→5.519ms in the prior repeat. No reproducible broad regression beyond runner variation is established, and no isolated speedup claim is made. Retain platform/per-mode measurement follow-up under this issue's existing deferral.

Final review disclosure: Claude/Composer reviewed c074b4d; the final direct finding fixes narrow native reuse to prefix and correct benchmark coverage/cleanup. Full local checks and local boundary review passed; the narrow final delta was not independently re-reviewed outside. Full-fuzzy native preparation remains deferred in issue54; exact-total collector strategy remains explicitly tracked in issue73. No npm publication or Harper pin change is included.

Comment generated by kAIle (GPT-6)

Harper adoption remains a separate gate

The current Harper 0.6.0 integration recognizes E_IDENTITY_MISMATCH and automatically rebuilds through its existing derived-index runtime. A local real 0.6.0→0.7.0 candidate probe passed, alongside 282 existing focused Harper checks (four conditional tests pending). No production pin was changed.

Adoption changes and the remaining valid-checkpoint, concurrent-write, multi-worker, crash/restart, packaged-artifact and rollback qualification are recorded in Implement full-text generation lifecycle, rebuild, eviction, backup, and status. That issue owns Harper rollout approval; wrapper qualification here does not satisfy it.

Comment generated by kAIle (GPT-6)

Harper-shaped relevance comparison

Paired 0.6.0/0.7.0 checks now cover 128 cases directly and through real Harper Table.search on audited RocksDB tables. All 80 synthetic catalog cases retained identical matching and ranked source IDs. Seven diagnostic stemmer cases changed ordered results (six membership changes and one fuzzy reordering), consistent with the ad versus add/added/adding/adds analyzer boundary. Tiny catalog score-bit changes did not alter rankings; four Harper highlight projections were identical. Forward/reverse insertion controls and 48 native pagination comparisons passed.

Full evidence, provenance and the remaining customer-judgment, durable-fixture, REST/permission, segment/scale and released-package criteria are tracked in Qualify native full-text filtering and catalog-scale operation for v5.4. This is local macOS arm64 / Node 24.16 synthetic evidence, not a production relevance-quality sign-off. Harper's 0.6.0 pin remains unchanged.

Comment generated by kAIle (GPT-6)

Reusable wrapper relevance fixtures

Upgrade fulltext to Tantivy 0.27 with safe index rebuilds now includes a runnable catalog example, shared synthetic records, explicit expected rankings and public-API regression runner. This adds durable wrapper coverage; it does not move Harper integration tests into fulltext.

• Run 92 query/profile cases in forward and reverse insertion order. Check ordered IDs, exact totals, versions, finite descending scores, UTF-8 ties and selected pagination. Include discriminating stop-word/synonym cases and approved ad/add stemmer differences without numeric score snapshots.
• Full local check passes: 74 Rust / 369 Node tests, plus packed-consumer and worker checks. All executable examples pass. Dropping stop-word/synonym options fails the new cases; published 0.6.0 passes the catalog cases and fails only the intentional stemming expectations.
• Claude and Composer reviewed the additions before narrow finding fixes. Full checks and a local second reader verified those final fixes; the 46-added/18-removed-line delta was not independently re-reviewed outside.
• Fresh supported-platform CI on the current example/test revision passes in final platform CI; production code is unchanged from the previously qualified benchmark head.

The relevance guide explains scope and maintenance. The historical paired driver, customer judgments, Harper REST/permissions/version races and scale/segment qualification remain in Qualify native full-text filtering and catalog-scale operation for v5.4. Exact-total collector strategy remains in Qualify Tantivy pruning for small-window ordinary ANY searches. No package publication or Harper pin change.

Comment generated by kAIle (GPT-6)