Add bounded deep-result traversal and qualify autocomplete limits

Read the wrapper contract in README.md, protocol limits in src/protocol.rs, and the public API in ts/native.ts alongside the engine in src/engine.rs. First establish the current limits and behavior, then design snapshot-aware traversal, expiration and reload/merge semantics, and benchmark common-prefix catalogs and candidate-filter workloads before proposing limit changes. Done means the documented traversal and limit policy satisfy the acceptance criteria, with tests demonstrating no omissions or duplicates for the promised snapshot.

Outcome

Add bounded deep-result traversal and qualify autocomplete limits.

Current behavior

Verified by source inspection of the public API and native implementation at the assessed revision; the initial assessment used source inspection, and the paging-window runtime reproduction is recorded below. The protocol caps offset+limit at 10,000, candidate IDs at 1,024, prefix expansions at 50, and autocomplete results at 100 with offset zero. These intentional bounds constrain export and broad autocomplete workflows.

Acceptance criteria

• Design snapshot-aware cursor/iterator traversal for results beyond the offset window, with explicit expiration and reload/merge semantics.
• Retain safe existing defaults and define whether autocomplete should have continuation or a separately qualified limit profile.
• Measure common-prefix catalogs and candidate-filter workloads before changing expansion/candidate/result ceilings.
• Reject overly broad requests explicitly rather than silently truncating ranking; test no omissions/duplicates for the promised snapshot.
• Document wrapper traversal as new work rather than claiming Tantivy provides a ready-made search-after API.

Harper integration and verified reproduction

Support full-text and autocomplete continuation in Table.search and REST tracks the dependent Harper API integration.

Observed through the built public Node/native API at the current base revision: 10,025 distinct records matched widget; offset: 9975, limit: 25 returned 25 hits, while offset: 10000, limit: 25 rejected with E_INVALID_ARGUMENT: search window must be between 1 and 10000.

The proposed implementation supports ranked ordinary, prefix, fuzzy-prefix, and Boolean continuation through searchPage() using a dedicated existing read-only handle to preserve the original snapshot while the writer publishes updates. The caller closes that handle after traversal. Legacy search() retains its 100-hit autocomplete limit; continuation uses the independent page-size and response-byte budgets. The exact-prefix 50-term expansion guard remains explicit. Regression coverage includes eligible hits beyond the first 100 raw matches, scored sequence equivalence, and retained snapshots during writes. Harper authorization, projection, and REST integration remain in the linked Harper issue. No release or merged implementation is claimed.

A separate 250-record all-widget published-0.5.0 runtime fixture confirms 250 prefix matches with only 100 returned hits, no continuation API, and rejection of the next offset. The local fix reaches all 250; the illustrative eligibility check accepts only IDs after the first 100. This is native runtime verification, not a live Harper reproduction.

The expanded audit is documented in the implementation’s limits guide. Separate query-policy configuration is tracked in Make query term and clause limits configurable with bounded resource policy.

References and scope

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API.

Wrapper contract and protocol limits. This is proposed standalone-library capability with no committed release. Application-specific authorization, source projection, and product API integration are outside this issue.

Qualification completed locally

The proposed implementation now includes a reproducible benchmarks/traversal.mjs qualification and a smaller platform-CI smoke profile. One optimized macOS arm64 / Node 24 run exercised 16,384 records, 12 actual segments, 3,906-byte ID payloads, 4,096-byte version payloads, and 76 fuzzy surface terms. These payload sizes are fixture inputs, not new limits; the ID ceiling remains 4,096 bytes. This mixed-vocabulary fixture has 216 exact-prefix matches, independently of the separate 250-widget reproduction above. Ordinary and fuzzy-prefix traversal returned all 16,384 original snapshot hits while writes published and ordinary searches completed alongside every page. Oversized responses rejected and smaller-page retries succeeded. These are local correctness/progress observations, not production latency or peak-memory guarantees; Harper/RocksDB integration is not measured.

The optimized admission test covers continuation retry after timeout, explicit queue saturation, ordinary-query capacity, close cancellation and successful reopening/search after releasing capacity. The full package gate now passes 250 Rust/Node/worker/consumer checks; two optimized concurrency scenarios pass separately for phrase search and traversal. Linux/Windows CI and publication remain pending.

Deferred work and owners

Deferred scope	Tracker
Term/clause configuration	Make query term and clause limits configurable with bounded resource policy
Lower-cost traversal	Reduce repeated matching work during deep ranked traversal
Direct Rust validation parity	Align direct Rust search validation with the public packed API
Higher candidate/expansion/payload profiles	Qualify higher candidate, prefix-expansion and payload capacity profiles
Analyzer/long-token/language changes	Support versioned per-field analyzers and additional languages
Fuzzy controls and matching/ranking policy	Make fuzzy edit distance and transpositions configurable
Harper rejection/coverage visibility	Implement full-text generation lifecycle, rebuild, eviction, backup, and status
Broader Harper aggregate-capacity qualification	Qualify native full-text filtering and catalog-scale operation for v5.4

The analyzer’s separately verified 40-byte omission boundary remains unchanged and is explicitly covered in the analyzer tracker. Direct Rust validation follow-up concerns legacy prefix bounds checked by decode_search rather than direct Engine::search; the protected Node boundary is not bypassed.

The Harper continuation integration still must bound its own reader leases and request work; deferring broader capacity qualification does not waive those requirements. Existing limits stay unchanged in this native traversal issue.

Corrective review verification

Implementation is proposed in Add bounded snapshot traversal for full-text and autocomplete results; the package remains unreleased.

The proposed implementation now authenticates complete continuation boundaries, including finite score/ID/snapshot edits and token splicing. Large identities use bounded asynchronous hash preparation, while small pages retain a bounded synchronous path and remain usable when large-hash capacity is full. Preparation rechecks command/byte admission after encoding, releases reservations before native submission, and checks deadline/close state after hashing. Existing accepted query-size bounds are unchanged.

Public regressions fail on the relevant pre-fix facade and pass after correction. Coverage includes concurrent initial tokens, real crypto-pool timeout, close, reentrant request encoding, capacity recovery, and independent phrase/traversal queued-timeout and native-saturation cases. The measurement harness and scoped before/after preparation observations are committed in the implementation; they do not claim an overall native-search speedup or production latency guarantee.

The preceding full review by Claude and Cursor Composer, with Harper domain adjudication, completed before these bounded follow-ups. The follow-ups now reject misplaced continuations and malformed requests with coded errors, restore independent literal pins for advertised limits, remove an unnecessary ordinary-path request copy, and test null/undefined continuation handling plus handle recovery. The remaining small encoder metadata allocation is an accepted tradeoff; the committed paired measurements do not resolve its cost.

The proposed implementation also checks snapshot expiry before snapshot-dependent query construction, preserving E_CURSOR_EXPIRED when newly indexed terms make an old prefix too broad. Traversal canonicalizes a copy of the candidate set, so order and duplicate counts do not change continuation identity. Original count/byte limits come from the native runtime and are checked before normalization; the original request must also fit the full-frame bound. Ordinary encoding is unchanged. Caller arrays and scoring field order are preserved. The collector reuses its immutable alive bitset, and CI smoke uses the default native budget while controlled full qualification retains its explicit budget.

Both new public writer/reader regressions failed before correction and pass after it, including Unicode/duplicate candidates, duplicate-count changes, mutation safety, changed-membership rejection, original full-frame bounds and exact/over-limit capacity boundaries. A separate negative control reproduces the duplicate-count rejection before the final correction. The exact local npm run check gate passes 250 package checks; both optimized concurrency scenarios and smoke/full traversal qualification pass. A reproducible same-query driver benchmark and qualified observations are committed locally. Remaining boxed-scorer overhead and repeated scans remain in the separate optimization tracker; no isolated-hoist speedup or production SLO is claimed.

The independent critical review also identified stale backend execution/cancellation documentation and the theoretical collision risk of a 64-bit snapshot fingerprint. The correction derives SHA-256 from an explicitly framed ordered segment/deletion map inside the engine, covering both direct Rust and Node callers without a cursor registry. Version-3 continuations carry 64-hex snapshots, remain authenticated with the existing object-local HMAC, and are bounded at 5,602 encoded characters. Earlier unreleased versions reject explicitly. Backend API, libuv preparation and traversal checkpoint documentation now match the implementation. Native SHA-256 vectors were independently calculated with Python, and existing retained-snapshot, tampering, old-format and capacity regressions pass. Cargo dependency policy checks pass and npm audit reports zero vulnerabilities. The SHA-256 follow-up also qualifies exactly 1/256 native segments and page sizes 1/32 against the pre-SHA engine, using three alternating optimized process pairs and complete scored-sequence/total assertions. Many-segment paired full-walk changes were +0.8–2.8% and +1.3–2.3%, respectively; those local costs are accepted for stronger intrinsic snapshot identity, not claimed as production bounds. Corrective review and a closing full review by Claude and Cursor Composer with Harper domain adjudication completed at 7bdb596, resolving the original findings and measured-performance follow-up. Subsequent documentation clarifies direct Rust 0.6 compatibility and permanent queue-capacity errors, and removes one redundant assertion; no production behavior changed, all 250 checks pass, and that narrow final delta was not independently re-reviewed. The enlarged header is an accepted documented change within the unreleased token format; bounded capability inspection and separately tracked production sizing remain accepted tradeoffs. The package remains unreleased. Platform CI must rerun on the updated PR head; Harper integration qualification remains pending.

Comment generated by kAIle (GPT-6)