Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search

Start by reading the public API in ts/native.ts, the native implementation in src/engine.rs, and the wrapper contract in README.md; compare their current query capabilities with Tantivy’s query documentation. Define bounded contracts and semantics for the requested query features, including interactions with filters, highlighting, and counts. Done means each capability has correctness and adversarial-cost tests, while raw parser syntax remains a separate API decision.

Outcome

Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search.

Current behavior

Verified by source inspection of the public API and native implementation at the assessed revision; no runtime reproduction was performed. The public text expression supports leaves and AND/OR/NOT, but has no clause boosts, required-with-optional scoring, regex, or MoreLikeThis request.

Acceptance criteria

• Define typed, bounded contracts for query-clause boosts (field-derived business signals belong to the native sorting/ranking issue) and required clauses with optional scoring clauses.
• Expose bounded regex and similar-document/field-value query capabilities, with source values supplied by the caller where necessary.
• Define scoring and field-selection semantics and interactions with filters, highlighting, and exact counts.
• Retain clause, expansion, response, and execution controls; add per-capability correctness and adversarial-cost tests.
• Keep raw query-parser syntax a separate API decision rather than requiring it for these capabilities.

References and scope

Metadata-derived business ranking is tracked in Expose native field sorting and bounded business ranking; this issue owns query-clause scoring.

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API.

Tantivy capability. Wrapper contract. This is proposed standalone-library capability with no committed release. Application-specific authorization, source projection, and product API integration are outside this issue.

Delivery sequence and request-level weights

Proposed sequence for implementing this issue; regex and similar-document acceptance remain in scope:

1. Deliver typed clause boosts and required clauses with optional scoring clauses first. Define optional-only queries, minimum optional matches, nesting, finite weight bounds and deterministic ties.
2. Explicitly assess per-request field weights as a separate control from existing index-definition weights. Define composition/precedence and whether they can be changed without rebuilding. This is proposed API work, not an existing capability claim.
3. Deliver bounded regex and similar-document queries as independent increments with their own cost and compatibility qualification.

Verify that scoring-only clauses do not change required-clause eligibility or exact totals. Preserve default scoring and analyze interactions with field selection, synonyms, phrases, prefix/fuzzy modes, tracing and pagination under current term/clause/deadline limits. Keep persisted metadata-derived business ranking with Expose native field sorting and bounded business ranking.

Harper query/REST exposure and authorization are owned by Expose full-text query boosts and optional scoring clauses through Table.search and REST. The wrapper retains the typed native scoring contract. Existing priority, effort, parent, type and milestone are preserved; no release is committed by this refinement.

Accepted composite scoring performance tradeoff

Add query boosts and optional full-text scoring lowers only root boosts directly over Any/All leaves into field weights, preserving native term-query pruning. Root boosts over composite OR/AND expressions retain native BoostQuery wrappers; those wrappers use a generic iterator instead of the specialized Boolean block-WAND path. Optional scoring with a required All branch also adds scoring work over the required-only intersection. This remains an accepted performance tradeoff for this delivery slice.

The paired 10,000-document optimized workload records complete request latency and verifies membership and score scaling. Repeated score ties can force stable-collector reruns in unboosted baselines, so these results dilute pruning differences and do not isolate the cost of losing block-WAND. A second varied-score corpus now asserts distinct ranks 20/21 for the paired OR/AND baselines, avoiding their boundary-tie rerun. The measured complete-request costs remain workload-specific; they do not count pruned blocks or establish catalog-scale behavior. Qualify larger representative corpora before deciding whether composite boost lowering is warranted. Preserve exact float32 score behavior and search/paging parity if changing that lowering.

Review-Deferral: 3084f89c82b555f2 severity=none category=performance
Review-Deferral: e33b81c1db8c0e17 severity=none category=performance

This note preserves the issue's existing priority, effort, parent, type, milestone and broader regex/similar-document scope.

Compatibility and release requirements

Add query boosts and optional full-text scoring deliberately rejects leaf-only fields and mode on Boolean groups rather than silently ignoring them. Its compatibility note also distinguishes rejected nested legacy operator: 'prefix' leaves from the retained top-level alias. Harper's compiler puts fields on leaves, so it does not depend on group-level properties. Retain this strict validation during rollout and call out the compatibility change.

Review-Deferral: e313f09f9410c6e0 severity=none category=correctness

Before publishing ABI 14/query API 8, select an unused package version and synchronize Cargo, the root facade, platform manifests and exact platform pins. Published 0.8.0 is ABI 13/query API 7. Runtime guards reject incompatible artifacts, but cannot make 0.8.0 publishable again. Harper's companion remains blocked on that wrapper release and its production pin/lock update. No version or release date is committed here.

Parallel ABI coordination

Source-verified comparison: Expose fulltext operation telemetry and latency histograms at its latest compared head claims ABI 14/query API 7 with telemetry API 1. Add query boosts and optional full-text scoring claims ABI 14/query API 8 without the telemetry contract. These are incompatible artifact sets under the same ABI number.

Before either contract is released, reconcile the native exports, runtimeInfo, facade types/copies, loader checks, tests and docs into one rebuilt artifact. If telemetry's ABI 14 ships first, scoring requires a fresh ABI and a matching Harper compatibility-pair update. Harper's scoring draft accepts 13/7 and 14/8; telemetry-only 14/7 deliberately rejects. This coordination blocks release, without assigning or publishing a version here.

Strict expression-shape compatibility

Add query boosts and optional full-text scoring intentionally rejects leaf-only fields and mode on Boolean groups with E_INVALID_ARGUMENT, including shapes older facades silently ignored. The normalizer at the reviewed revision fails closed rather than discarding these properties. Callers must place text options on text leaves. The README compatibility contract documents the breaking validation change and matching ABI 14/query API 8 requirement. Retain this validation during artifact reconciliation and rollout; this marker records an intentional contract change, not a new defect or a promise to restore ignored properties.

Review-Deferral: e5b36a05fc329c13 severity=none category=correctness

Package-version release gate tracking

The facade and exact platform pins still reuse published 0.8.0 while this draft requires ABI 14/query API 8. This is the existing publication gate: coordinate an unused package version, matching native artifacts and exact platform pins with the parallel telemetry ABI work before release. Retain the strict-shape compatibility contract above. No new issue, priority change or release version is assigned by this tracking note.

Review-Deferral: 8703f3426d90022d severity=none category=reliability

Comment generated by kAIle (GPT-6)