Qualify remaining native metadata collector resource tradeoffs

Start with the PR #75 benchmark guide and its reproducible drivers, then inspect the native metadata collectors and the evaluation matrix in this issue. Record pinned wrapper and Tantivy revisions, fixture parameters, matched outputs, CPU, persisted bytes, and process memory across the listed platforms and concurrent cold-request cases. For each policy, document evidence to retain it or a bounded native alternative; do not change release or persisted-format behavior.

Outcome

Maintain a reproducible qualification baseline for the deliberate resource and performance choices in Add global metadata sorting, facets, and filter predicates, under Epic: Extend fulltext search capabilities beyond the initial wrapper API.

Selective sort admission, active Count cancellation, completed dictionary caching, and the missing correctness regressions are addressed in that PR. This task evaluates further native reuse and capacity improvements; it does not reopen those fixes or assume every current policy needs replacing.

Linux sustained-memory follow-up

Decision: prioritize production Linux. Treat this work as nonblocking unless evidence demonstrates accumulating retention or inadequate production memory headroom. This does not waive other correctness, cancellation, shutdown, migration or review requirements for Add global metadata sorting, facets, and filter predicates.

• Predeclare a longer-duration Linux protocol with repeated mixed-load/drain cycles, fixed dataset and concurrency, a pinned build, duration, measurement windows and memory-headroom criteria before running it.
• Compare equivalent post-drain points across cycles for resident set size (RSS), V8 used/physical heap, and allocator in-use/free-retained memory. Retain startup telemetry, garbage-collection (GC) observations and raw samples; disclose instrumentation overhead and overlapping accounting.
• Require correct query results, latency/throughput gates, queue recovery and final checkpoint/reopen verification. Preserve failed-run evidence separately from successful acceptance paths.
• Assess whether post-drain baselines stabilize or accumulate across cycles and whether measured peaks fit the declared Linux deployment budget. A finite passing run establishes only the tested duration and workload.
• If evidence supports changing the symmetric RSS-span criterion, propose the replacement and rationale explicitly. Keep the original 32 MiB gate results; do not retroactively relabel either failed run as passed.
• Record the result and residual uncertainty here. Escalate to a blocking implementation investigation if sustained retention or inadequate headroom is demonstrated.

Open items: the longer repeated-cycle Linux experiment has not run; deployment memory headroom and quantitative stability criteria still need to be specified. The observed V8/RSS correlation does not establish cause or leak absence. macOS attribution remains separate and is not a production-readiness prerequisite.

Evaluation matrix

Current policy	Work to qualify before changing it
Deadline adapter traverses native scorers	Compare cancellable native counting/traversal options with current cooperative checkpoints. Preserve active interruption; segment-only deadline checks are insufficient. Record bare Count and end-to-end exact-total costs separately.
Unsafe histogram segments validate matching values	Compare the existing rare fallback with native metric siblings. Include safe segments, unmatched/deleted extremes and multivalue buckets; preserve rejection instead of saturated buckets.
Concurrent cold dictionary misses may repeat scans	Measure cold-miss amplification and bounded coordination/resumption options without making one request wait on another's deadline. Publish only complete maxima.
Every string filter has a raw fast column	Measure equality-only storage/ingestion alongside sorting/facets. Any capability flag needs an explicit public API and identity/migration design.
Broad fragmented sorts can exceed the retained-key budget	Evaluate bounded incremental native fruit merging, including native aggregation intermediates. Preserve global scoring statistics, ordering, exact totals and terms uncertainty; never merge finalized averages or prune intermediates silently.
Writer string deduplication is retained	Extend scalar/unique/repeated-array qualification across architectures. Native terms already deduplicates document counts; retain the set only on its measured storage/read benefit, preserving first-value sorting.

Qualification completed

Warm concurrent cache access was measured on eight in-memory indexes with 40 segments each. Three alternating local macOS arm64 baseline/candidate pairs did not meet the declared replacement threshold: eight-caller preflight median p50 was identical for Mutex and RwLock. The mutex remains. The retained baseline preflight p50 rose 3.73× from one to eight callers; this comparison does not attribute the increase to lock waits versus fast-field work, memory contention or scheduling. Attribution and cross-platform qualification remain open; the public cold-start experiment below qualifies first-search behavior without counting duplicate scans. See the reproducible fixture and limits in the PR's benchmark guide.

The sort-budget fallback's Count pass supplies the retained-hit bound only when conservative estimates would reject. Unsorted aggregation requests retain ranked-hit collection because search() returns hits; exact counting shares the aggregation pass. These are accepted behaviors, with future alternatives covered by the evaluation matrix above.

Additional local qualification, 2026-10-28

Three alternating optimized previous/current process pairs used 100,000 documents and 1/4/8 closed-loop callers. Each version completed 2,160 normal exact-count requests correctly; 1,080 budgeted AND requests succeeded and 1,080 budgeted OR requests returned only E_TIMEOUT. All recovery batches returned correct totals with zero queued commands/bytes. At eight callers, AND throughput changed −8.2%, OR −1.7%; p99 changed +7.3% and +4.4%. These are whole-branch local comparisons, not isolated Count costs or production service objectives.

Fresh-process dictionary-cold cohorts used eight indexes with 65,536 distinct strings each, shared and independent targets, and 1/4/8 callers. Across three rounds, first-wave p95 was at most 3.714 ms. All normal requests returned exact total 16 and 16 one-document buckets; 62 of 78 one-millisecond requests timed out, and every subsequent recovery succeeded. Filesystem caches were uncontrolled and likely warm. CPU/RSS were recorded; duplicate-scan attribution and sustained memory stability were not established.

Decision: retain cooperative Count and the existing mutex/cache admission. These local results do not justify another runtime mechanism. The PR's benchmark guide contains reproducible drivers and measurement tables; Linux/Windows performance, disk-cold startup, larger/fragmented dictionaries, sustained memory behavior, and application-specific latency budgets remain open.

A genuine previous-version string-index reject/reset/rebuild/publish/reopen qualification also passed, including unchanged old file contents on rejected opens and numeric/Boolean compatibility. The tested migration guide keeps application cutover, catch-up and rollback qualification with the consuming application. Kyle approved the API and migration recommendations on 2026-10-28. Subsequent multi-field sorting and nested aggregations retain the same persisted string identity and rebuild procedure.

Publication prerequisite

This draft remains unreleased and still uses the previous package version. Before publication, choose a new version and update the root package, Cargo package, exact optional platform dependencies and lockfiles together. Installing or packing a source-only metadata facade against the published ABI-9 0.7.0 platform package will fail closed at ABI validation; it is not an install-ready release. The publish workflow runs on a published release and checks tag/manifests. No version, tag, release or consumer pin is changed by this qualification task.

The migration qualification additionally passed old-build rejection of the new string index without mutation, a new numeric/Boolean publication recovered by the old reader, and reset/rebuild downgrade reconstruction. Live cutover/rollback under production traffic remains application work.

Native expansion resource policy

Multi-field sorting and nested aggregations remove the fixed eight-aggregation and 100-term/range caps. Large requested terms sizes remain valid when actual cardinality is small. Static request, construction and depth checks run before collection; native memory and global bucket guards, decoded-key admission and bounded response encoding govern actual data-dependent growth. A selective or empty query can therefore succeed where the same shape over more matching data rejects. Future qualification may compare bounded native alternatives, without restoring arbitrary feature caps or duplicating full aggregation collection.

Premerge consumer qualification

The local synthetic consumer rehearsal now uses actual built Harper Table/Resource writes with a durable harness-owned journal. Snapshot checkpoint 7 is captured before reading rows and catches three mutations before/during indexing through 10, including idempotent replay of an update already visible in the snapshot; publication/reopen, resumed writes through 11, and old-build reconstruction/reopen pass. This tests sequential local interleaving, not distributed replication, an HTTP service or a deployed fulltext adapter. The installed published 0.7.0 optional-native-package upgrade/downgrade route also passes.

Qualification exposed a native-handle shutdown defect: Tantivy rollback replaces the writer, so waiting on its replacement leaves original merges outside the close barrier. The PR now closes the original writer directly. A gated production-path regression fails before and passes after while preserving committed records/checkpoint and discarding pending upserts/deletes. A separate 100,000-record concurrent count run recovered after all 180 budgeted dense-OR requests returned E_TIMEOUT. Native-handle shutdown and reusable Rust rollback are distinct: the public Rust rollback primitive still replaces its writer and is not a quiescence barrier; terminal Rust shutdown must consume close directly. A stronger reusable Rust rollback ownership contract remains follow-up work. A separate natural two-worker commit-preparation failure exposed detached worker handles; the PR now preserves an unproven-ownership flag so later close cannot falsely release the path reservation. The regression fails before and passes after this fix.

Three final alternating same-feature baseline/current process pairs passed unchanged gates: worst median p95 ratio 1.003 and lowest throughput ratio 0.995. A six-minute expanded synthetic catalog run completed 895,680 normal requests across 12 cells, with worst pooled p95 10.143 ms, maximum sampled RSS growth 45.125 MiB and maximum last-five-round span 2.969 MiB.

An earlier flat run exceeded the 32 MiB tail-span gate; its direction and cause remain unknown because sufficient samples were not retained. Ten traced and ten untraced fresh processes subsequently passed unchanged gates (maximum span 0.969 MiB). Retain this unexplained observation for merge review and future memory qualification; passing reruns do not establish leak absence. Discard close also drains unpublished indexing work and requires storage/time headroom; a natural index-worker storage-failure regression verifies unproven quiescence is reported rather than allowing unsafe directory reuse.

Version selection remains the release step at Kyle's explicit request. Workload/RSS acceptance is recorded separately in the PR's premerge guide; no production performance objective is established by these local checks.

Mixed soak qualification, 2026-10-29

Historical macOS qualification failed; retained as development-platform evidence. The 40-minute synthetic product-catalog soak (30 minutes mixed reads/writes, ten minutes after writer drain) completed 2,823,976 oracle-checked queries, 901 publications and 108,110 measured mutations. Worst phase/case p95/p99 was 8.1/9.0 ms; throughput was 1,175.863 queries/second. Final sampled queues were empty, writer drain succeeded, and failure cleanup reported no close errors. A separate retained-index reopen verified checkpoint soak:901, four scenarios and hit versions against 49,990 reconstructed source records.

The run failed its unchanged 32 MiB final-five-minute RSS-span gate: mixed span 114.375 MiB, drain span 38.969 MiB. Sampled peak growth was 128.422 MiB, below the separate 256 MiB growth gate. The mixed excursion includes a sharp RSS drop/rebound; its window endpoint change was +2.141 MiB. The drain window ended +37.156 MiB above its start. The symmetric span gate detects either direction; this run does not establish a leak or leak absence. It does not explain the earlier short-run failure whose samples were insufficient.

Historical investigation proposal: isolated macOS process physical/compressed footprint and allocator/GC observations from startup. This is now a separate development-platform follow-up; production Linux qualification below takes priority. Distinguish retained allocation from resident-page changes before proposing a different memory criterion. Preserve the original failed result and unchanged gate; do not treat a passing rerun as its explanation. Later host telemetry observed substantial swap/compressed memory but began after the mixed excursion, so causation remains unproven.

The failed driver's normal final-reopen/removal path was not reached. Throughput, the drain span and final queues were calculated independently from raw samples; the checkpoint reopen was a separate post-failure check. Raw samples, failure diagnostics and the retained native index remain in local scratch. The committed measurement summary records values and artifact fingerprints; the premerge guide records limitations and reproduction. This local workload is not a Harper service or production performance qualification.

Windows CI qualification remains open

The stage-instrumented run hit the unchanged 60-second workload child deadline on Windows Node 22 and 24. Query cases completed in 1.1–1.3 seconds; long-key preparation reached 35.9–38.6 seconds, then native writer close remained unfinished. One Node 22 concurrent inspection/publication test additionally returned E_STORAGE with access denied writing a fieldnorm file. Neither cause is proven.

The controlled scheduling experiment reserves a serial batch for the two deadline-bound qualification files, retaining fixture, debug build, child deadlines and gates. The full local isolated-npm gate passed 126 Rust and 422 Node/worker/package tests across five groups. Compare Windows ingestion/close profiles before attributing contention. A green isolated run or absent access-denied error does not establish either cause or resolution. Keep draft and preserve the separate failed long-soak RSS qualification.

Current Windows CI recurrence — 29 October 2026

Make analyzed query capacity configurable per handle exposed two distinct failures in the initial expanded-policy run:

• Windows Node 22 failed a metadata test cleanup hook with ENOTEMPTY. Its fixture registered directory deletion before writer close; Node runs these hooks in registration order. The same ordering exists on main, and a local closed-state assertion fails deterministically with state open before deletion. The test-only ordering fix awaits successful close before deleting and preserves the directory on cleanup failure. All 20 focused metadata/capacity cases pass locally. The subsequent current-head CI run passed all 14 checks, including both Windows jobs, with two draft review jobs skipped. This does not establish the separate access-denied cause or its resolution. No runtime, retry or timeout change is included.
• Windows Node 24 separately failed the mixed-soak writer-cleanup test: mixed work returned access-denied E_STORAGE before the expected injected E_CLOSE_FAILED. The injected cleanup error was retained separately. The storage error's native cause remains unclassified; the log does not identify the affected file. This is further Windows storage evidence, with no proven relationship to the fixture-ordering bug or query-capacity changes.

Both jobs also passed all 134 Rust tests (five ignored) and the larger-tree/count-validation regressions. Preserve both failures and the unchanged oracles. A green subsequent run establishes neither the access-denied cause nor its resolution. Linux production sustained-memory qualification remains a separate follow-up.

Comment generated by kAIle (GPT-6)

Node 22 recurrence and diagnostic follow-up — 30 October 2026

Observed: Windows Node 22 at the refinement head failed the short mixed-soak success test with access-denied E_STORAGE after 194 correct queries, 110 mutations and zero completed mixed publications. Thirteen checks passed, including Windows Node 24; two review checks were skipped. The driver cannot distinguish commit preparation, final publication and replacement-reader installation from this historical report; its native error has no filename. This does not establish a query-capacity regression or resolve the earlier Windows failures.

Verified source constraints: pinned Tantivy 0.27 preserves its managed-file lock across registration writes. The specific race alleged in ManagedDirectory commit fails with PermissionDenied on Windows is disputed by its source/reproduction follow-up and is not established here. Atomic metadata writes can return an I/O error without a filename; registration can report a segment filename even when the earlier managed-metadata write failed. No specific native file or external locker has been identified.

Diagnostic work: Make analyzed query capacity configurable per handle now adds Node 26 on all four existing CI platforms, identifies the first failing soak operation, distinguishes Tantivy preparation from final commit in error context, and preserves secondary final-reader close errors. CI retains only the approved synthetic fixture manifest, failure report and samples for 14 days after failure; native indexes and memory maps are excluded. Native error codes and original messages remain intact. No retries, workload reductions, deadline increases or relaxed acceptance gates are included.

Observed CI: The expanded matrix at the diagnostic revision passed all 18 checks, including Windows Node 22.18.0, 24 and 26. The two draft review checks were skipped. This produced no new Windows failure artifacts and does not establish the historical storage error’s cause or resolution.

Open: inspect current-head Windows diagnostics if the failure recurs, identify the failing native operation/file and reproduce its owning invariant before proposing a runtime fix. A green run alone does not establish cause or resolution.

Comment generated by kAIle (GPT-6)

Acceptance

• Record pinned wrapper/Tantivy revisions, repeatable fixture parameters and matched native outputs.
• Cover Linux x64/arm64 and the supported desktop platforms where available, plus concurrent cold requests and multiple indexes.
• Measure CPU, persisted bytes and process-memory observations separately; do not equate native accounting limits with peak RSS.
• For each policy, retain it with evidence or propose a bounded native alternative. No release commitment is implied.

Priority P3: follow-up qualification in an uncommitted capability backlog, with no demonstrated customer outage. Effort Medium: benchmark and collector work spans the native subsystem; no production or persisted-format change is authorized by this task alone. No customer or security incident produced this work.

Review tracking

Review-Deferral: 51710a2cbff341ec severity=minor category=performance
Review-Deferral: 13ced9d76f166c4a severity=nit category=performance
Review-Deferral: 60bd37ffc115006f severity=nit category=performance
Review-Deferral: a4c3187707f037c2 severity=nit category=performance
Review-Deferral: e85a0d28f4afaf33 severity=nit category=performance
Review-Deferral: a5a9756310ad2731 severity=none category=reliability
Review-Deferral: 6d9e4fcbfef0c303 severity=nit category=performance

Review-Deferral: ae818e3db867fa22 severity=none category=performance
Review-Deferral: 93475d831091d1ef severity=none category=performance

Review-Deferral: 9aa9e453eec840a4 severity=nit category=performance
Review-Deferral: 327385317e1135ec severity=nit category=performance

Review-Deferral: 734c4f147485383e severity=nit category=performance

Review-Deferral: 70f280ddf74a224d severity=none category=test-coverage

Review-Deferral: 748df37fecd5cef4 severity=none category=reliability

Review-Deferral: 2b64bb0181ef55de severity=none category=performance
Review-Deferral: 11a5564f469e8802 severity=none category=reliability
Review-Deferral: 48bdbfe7004f849b severity=none category=reliability

Review-Deferral: 9482131cd3ab1c16 severity=nit category=reliability
Review-Deferral: dd59793efadb43fe severity=nit category=performance
Review-Deferral: 619bbb7d07d9e7f5 severity=nit category=performance

Review-Deferral: a4995f2c0c16770e severity=minor category=performance
Review-Deferral: fbd700da9cb9cff2 severity=nit category=reliability

Review-Deferral: bb6d3bf663e9162e severity=nit category=maintainability

Paired Linux memory evidence, 2026-10-29

Linux sustained-memory qualification remains a nonblocking follow-up; no production memory defect has been demonstrated. Both runs used the unchanged default 40-minute synthetic workload at the qualified revision, with the original gates and no forced GC or allocator/OS tuning.

Observed result	External-only control	Instrumented comparison
Oracle-checked queries	1,603,967	1,579,392
Publications / mutations	901 / 108,110	901 / 108,110
Mixed final-five-minute RSS span	8.000 MiB	8.586 MiB
Drain final-five-minute RSS span	12.969 MiB	40.625 MiB: failed 32 MiB gate
Peak RSS growth	140.570 MiB	145.680 MiB
Worst phase/case p95 / p99	13.9 / 15.7 ms	15.1 / 16.1 ms

Verified independently from retained samples: the control passed; the comparison failed its drain RSS-span gate. Latency and throughput passed in both. The failed comparison retained its index and reported no close errors; its successful final-checkpoint acceptance path was not reached. It must not be relabeled accepted.

Verified instrumentation completeness: 2,404 paired sample/cost sequences, a final exit record, and no sampler errors. Raw p99 allocator-call / whole-sample costs were 0.321 / 0.650 ms, below the predeclared 2 ms timing-validity threshold. Whole-sample cost includes the primary write but excludes writing its own cost record. Low sampling cost does not establish absence of allocation or GC perturbation.

Observed in the same runtime records at the instrumented drain-tail RSS extrema: RSS rose 168.336 → 208.961 MiB while V8 physical heap rose 35.422 → 76.395 MiB. Glibc in-use bytes changed 5.568 → 6.395 MiB, and free-retained bytes 67.432 → 66.605 MiB. These measurements are sequential, not atomic. This supports investigating V8 heap/GC behavior, without establishing causation, probe responsibility, or leak absence. Glibc counters cover the whole process, overlap V8 external allocations, and cannot identify Tantivy ownership.

Observed maximum index-file resident memory was 4.324 MiB in control and 4.031 MiB instrumented; process swap remained zero. Instrumented host counters recorded 190 swap-in pages, zero swap-out, and no increments in pgscan_kswapd, pgscan_direct, or oom_kill; this does not establish absence of all reclamation.

Private raw artifacts and independent analyses are preserved locally and remotely. A second-reader artifact check corrected residency scope, reclamation wording and measurement simultaneity. Linux evidence cannot explain Darwin compression/residency or close the original failed macOS gate. Kyle clarified that macOS is not used in production; resolving that observation is a separate development-platform follow-up and does not by itself block merge. Preserve both failed measurements and the original gates; no release or merge action follows from this issue update.

Query-control construction admission

Distance-two fuzzy matching charges four construction work units per eligible term/field against the existing 256-unit request budget. The weight derives from sampled native reachable-DFA-state ratios; it is not a hard process-memory bound. Large concurrent construction failed the unchanged 256 MiB local RSS-growth gate, including the saved distance-one baseline. The verified finite comparison and repeated controls retain all failures; a single passing repeat does not qualify the workload generally.

• Include long eligible terms and distance-two native construction in the production-Linux repeated-cycle/headroom protocol above. Separate per-request construction, concurrent peaks and post-drain retention before considering a policy change. No speculative allocator workaround, new claim of a hard bound, or relaxed gate is implied.

Retain the existing P3/Medium investigation and no milestone: review severity is major performance, while production impact and insufficient Linux headroom have not been demonstrated. This does not erase or downgrade the failed qualification evidence.

Review-Deferral: 06bf5933d3b05e71 severity=major category=performance

Comment generated by kAIle (GPT-6)

Query-controls review follow-up

Distance-two preview has no per-handle operator ceiling or hard native-memory bound. Keep the failed construction/RSS measurements and qualify production headroom before exposure; this PR does not add an arbitrary wrapper cap. See Expose fuzzy matching controls and align Rust request validation.

Review-Deferral: 7ad5cd4522b851ce severity=major category=performance

Comment generated by kAIle (GPT-6)