Expose low-overhead fulltext observability and operation latency metrics

Start with the queue and cumulative timing counters in src/native.rs and the public status shape in ts/native.ts, to see what is already reported. The histogram API, sampling defaults and overhead budget are undecided and need qualification evidence first, so this is design work rather than a first task. Done means the agreed metrics appear in status output with documented units and passing overhead and counter-conservation tests.

Current implementation and remaining acceptance

Expose fulltext operation telemetry and latency histograms #81 implements the wrapper contract. It remains draft and unreleased; this issue records delivery evidence, retained tradeoffs and publication prerequisites rather than production qualification or merge approval. The original proposal is preserved below as historical context; its directory-scan and active-cancellation suggestions are superseded by the explicit supported/unavailable contract.

Implemented in the draft

• Native admission, queue occupancy/age/capacity, dispatched work, outcomes, fixed errors, request bytes and queue/execution totals; independently opt-in histogram distributions.
• Commit, reader reload and teardown observations; facade logical-batch/partial-frame and preparation-rejection accounting, with overlapping metrics explicitly distinguished.
• Reader-visible document/deletion/segment/generation observations using Tantivy's supported APIs. Optional reader-snapshot file bytes carry pending, age and error state. No directory walk on scrape, automatic-merge reconstruction, per-index RSS estimate or active-cancellation API is introduced.
• Public writer/reader telemetry() and configureTelemetry(), strict configuration and explicit unavailable results. Scalar final observations survive close without keeping index resources alive. Sequential facade rewrap inherits native settings; concurrent owners remain unsupported.
• Caller-owned opt-in collection and executable pause/resume example. Basic counters/timing always accumulate; histograms and byte observations start disabled. Polling can stop/resume on existing handles without reset or reopen; no wrapper monitoring timer or SDK is added.

Contract and evidence: wrapper observability guide, qualification report, native accounting implementation and public writer/reader controls.

Evidence and limits

The full-check runtime revision passed 184 Rust tests (nine ignored), 535 Node/worker/package tests and twelve optimized tests. Later opt-in documentation/example/test changes passed both production-addon writer/reader controls, all ten examples and formatting; the full runtime suite was not rerun for them. Do not relabel the historical full-suite result as exact-final validation.

The same-build collection comparison validated 28 million timed searches. At roughly one staggered call/second per handle, median throughput cost was 0.82%/1.24% for four/sixteen indexes, with a worst paired loss of 5.75%; basic recording stayed active in both profiles. Earlier worker-banking measurements, all raw hashes, noisy ordinary-search cells and failed memory gates remain preserved. These local synthetic macOS observations establish neither a universal overhead bound nor production Linux/sustained-memory qualification. Source-only and optional platform package identities must remain distinguished.

Retained tradeoffs and follow-up triggers

Accepted contract clarifications

• Before native query classification succeeds, failures belong to search; no reliable expense class exists after classifier panic. The shared production selector and ordinary/expensive/unknown-class regressions pin the documented fallback. This is not a promise to reclassify a panicked request.
• snapshotBytesPending means a measurement is needed; it does not promise queued/running work. Enabling alone schedules no work. A subsequent successful publication/reload gives an observation opportunity. Documentation and public controls explicitly retain this distinction.

Independent adjudication retains these contract decisions as accepted-known, rather than claiming behavior was changed. Historical markers remain searchable at their original severity/category; documented behavior is not presented as a newly discovered defect.

Review-Deferral: 0f8522346a859f88 severity=nit category=correctness
Review-Deferral: 2335b89f8c04c2bf severity=nit category=performance
Review-Deferral: 488edead300a34ba severity=nit category=performance
Review-Deferral: 571ef85f754f945e severity=nit category=correctness
Review-Deferral: 657d962ab7f02c8a severity=nit category=maintainability
Review-Deferral: f8bb6c1668f0d6b6 severity=major category=performance
Review-Deferral: 259bce1b9a03dd27 severity=nit category=performance
Review-Deferral: b679793ce2beb276 severity=nit category=maintainability
Review-Deferral: 756fec0c94d539dd severity=nit category=performance
Review-Deferral: 7f007587637804a2 severity=minor category=reliability
Review-Deferral: d1820981df997465 severity=nit category=reliability
Review-Deferral: 06c2d5a68305635a severity=nit category=performance

The fresh managed review at df54b0b grades f8bb6c1668f0d6b6 as major performance under its hot-path floor. This updates the review classification of the existing accounting-cost follow-up; it does not add a production measurement or revoke the accepted always-on accounting tradeoff.

Publication and consumer prerequisites

• Select a release version and align root/Cargo/platform manifests, exact optional dependencies, lockfiles and release artifacts before publishing. Current source requires native ABI 14 / telemetry API 1; published 0.8.0 platform binaries have ABI 13. Coordinate any intervening ABI change from Add query boosts and optional full-text scoring #82. No version, tag, release or consumer pin is selected by this reconciliation.
• Reconcile retained/retired review identities through the standard managed CLI: all 13 current deferrals have verified open issues and the managed filing gate passes. Keep this issue open for retained work and the publication prerequisite.
• Implement and qualify Harper collection separately in Implement full-text generation lifecycle, rebuild, eviction, backup, and status #2510. Keep source freshness, authorization and end-to-end attribution Harper-owned, and production workload gates in Qualify native full-text filtering and catalog-scale operation for v5.4 #2512.

Review-tool limitation: the full primary review traced the example awaits but cited three lines with same-file shorthand (:41, :42, :43). The validator reports those references as missing. A format-only diagnostic expanding the same citations changes validation from incomplete to valid; the independent report and receipts were not modified. Both variants retain seven groups beyond the displayed scan cap. The tool's coverage warnings remain visible and formal review convergence is not claimed; this limitation is tracked separately. No source code, deadline, test oracle or release version changed during reconciliation.

Search preparation error normalization stays in Align direct Rust search validation with the public packed API #65. Broader sustained memory and collector qualification stays in Qualify remaining native metadata collector resource tradeoffs #76. Existing priority/effort/assignment/milestone remain unchanged; no new issue or release commitment is introduced.

Comment generated by kAIle (GPT-6)