Expose low-overhead fulltext observability and operation latency metrics

Aperta
  • Qualcosa non va?
#62 3 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@kylebernhardy ci sta già lavorando.

Dal 31/10/2026.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust, typescript
Ambito
observability, search

Direzione di ricerca

Start with the queue and cumulative timing counters in src/native.rs and the public status shape in ts/native.ts, to see what is already reported. The histogram API, sampling defaults and overhead budget are undecided and need qualification evidence first, so this is design work rather than a first task. Done means the agreed metrics appear in status output with documented units and passing overhead and counter-conservation tests.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Current implementation and remaining acceptance

Expose fulltext operation telemetry and latency histograms #81 implements the wrapper contract. It remains draft and unreleased; this issue records delivery evidence, retained tradeoffs and publication prerequisites rather than production qualification or merge approval. The original proposal is preserved below as historical context; its directory-scan and active-cancellation suggestions are superseded by the explicit supported/unavailable contract.

Implemented in the draft
  • Native admission, queue occupancy/age/capacity, dispatched work, outcomes, fixed errors, request bytes and queue/execution totals; independently opt-in histogram distributions.
  • Commit, reader reload and teardown observations; facade logical-batch/partial-frame and preparation-rejection accounting, with overlapping metrics explicitly distinguished.
  • Reader-visible document/deletion/segment/generation observations using Tantivy's supported APIs. Optional reader-snapshot file bytes carry pending, age and error state. No directory walk on scrape, automatic-merge reconstruction, per-index RSS estimate or active-cancellation API is introduced.
  • Public writer/reader telemetry() and configureTelemetry(), strict configuration and explicit unavailable results. Scalar final observations survive close without keeping index resources alive. Sequential facade rewrap inherits native settings; concurrent owners remain unsupported.
  • Caller-owned opt-in collection and executable pause/resume example. Basic counters/timing always accumulate; histograms and byte observations start disabled. Polling can stop/resume on existing handles without reset or reopen; no wrapper monitoring timer or SDK is added.

Contract and evidence: wrapper observability guide, qualification report, native accounting implementation and public writer/reader controls.

Evidence and limits

The full-check runtime revision passed 184 Rust tests (nine ignored), 535 Node/worker/package tests and twelve optimized tests. Later opt-in documentation/example/test changes passed both production-addon writer/reader controls, all ten examples and formatting; the full runtime suite was not rerun for them. Do not relabel the historical full-suite result as exact-final validation.

The same-build collection comparison validated 28 million timed searches. At roughly one staggered call/second per handle, median throughput cost was 0.82%/1.24% for four/sixteen indexes, with a worst paired loss of 5.75%; basic recording stayed active in both profiles. Earlier worker-banking measurements, all raw hashes, noisy ordinary-search cells and failed memory gates remain preserved. These local synthetic macOS observations establish neither a universal overhead bound nor production Linux/sustained-memory qualification. Source-only and optional platform package identities must remain distinguished.

Retained tradeoffs and follow-up triggers
Review topic Current disposition / future trigger
Always-on accounting Retained design with measured costs. Tight 1%/3%/5% limits remain unestablished; the initial acceptance covers the original recorder, with later banked-recorder ordinary-cost uncertainty separately disclosed. Revisit if matched production workload measurements demonstrate material cost.
Worker-scaled snapshots Aggregation visits configured worker banks. Measurements cover 1/4/16/64 workers and public collection; they do not provide a loaded pause bound. Stagger collection and qualify the actual consumer schedule.
Error-counter sharing Worker failures now use stable worker banks; admission rejections and unbound drains retain shared fallback counters. Failure/success and mixed producer controls are recorded. Measure reachable public workloads before changing serialized admission ownership or promising contention-free failure storms. Both historical path-based identities below refer to this reconciliation.
Partial configuration Omitted options merge from a native-authority snapshot, incurring cold-path collection cost; full explicit configuration skips that read. No snapshot is read on search/batch paths. Optimize only with a measured need and preserved ownership semantics.
Scrape-driver cleanup The isolated idle scraper uses rollback during fixture cleanup, including successful measurements. This is a recorded harness choice, not production close behavior; the paired workload driver requires clean close on success. Any future change must preserve primary/secondary errors and verified fixture removal.
Benchmark invocation The paired benchmark consumes separately prepared absolute facade/addon modules and intentionally does not rebuild them. Follow the reproduction commands; evaluate a CLI convenience change without silently replacing either measured artifact or hiding mismatched ABI failures.
Writer clock reads Retain the distinct writer operation/legacy timing boundaries, including close teardown. A clock-consolidation change must preserve those spans and be justified by isolated measurements; current mutation/publication controls do not isolate this small cost.
Soak publication minimum Retain the parent child deadline and unchanged workload/oracles. The loop already stops on its first error; a slow successful iteration can still reach the child deadline before satisfying a requested minimum. Improve finite-fixture diagnostics only with a natural regression, without retries, timeout increases or relaxed acceptance.
Accepted contract clarifications
  • Before native query classification succeeds, failures belong to search; no reliable expense class exists after classifier panic. The shared production selector and ordinary/expensive/unknown-class regressions pin the documented fallback. This is not a promise to reclassify a panicked request.
  • snapshotBytesPending means a measurement is needed; it does not promise queued/running work. Enabling alone schedules no work. A subsequent successful publication/reload gives an observation opportunity. Documentation and public controls explicitly retain this distinction.

Independent adjudication retains these contract decisions as accepted-known, rather than claiming behavior was changed. Historical markers remain searchable at their original severity/category; documented behavior is not presented as a newly discovered defect.

Review-Deferral: 0f8522346a859f88 severity=nit category=correctness
Review-Deferral: 2335b89f8c04c2bf severity=nit category=performance
Review-Deferral: 488edead300a34ba severity=nit category=performance
Review-Deferral: 571ef85f754f945e severity=nit category=correctness
Review-Deferral: 657d962ab7f02c8a severity=nit category=maintainability
Review-Deferral: f8bb6c1668f0d6b6 severity=major category=performance
Review-Deferral: 259bce1b9a03dd27 severity=nit category=performance
Review-Deferral: b679793ce2beb276 severity=nit category=maintainability
Review-Deferral: 756fec0c94d539dd severity=nit category=performance
Review-Deferral: 7f007587637804a2 severity=minor category=reliability
Review-Deferral: d1820981df997465 severity=nit category=reliability
Review-Deferral: 06c2d5a68305635a severity=nit category=performance

The fresh managed review at df54b0b grades f8bb6c1668f0d6b6 as major performance under its hot-path floor. This updates the review classification of the existing accounting-cost follow-up; it does not add a production measurement or revoke the accepted always-on accounting tradeoff.

Publication and consumer prerequisites

Review-tool limitation: the full primary review traced the example awaits but cited three lines with same-file shorthand (:41, :42, :43). The validator reports those references as missing. A format-only diagnostic expanding the same citations changes validation from incomplete to valid; the independent report and receipts were not modified. Both variants retain seven groups beyond the displayed scan cap. The tool's coverage warnings remain visible and formal review convergence is not claimed; this limitation is tracked separately. No source code, deadline, test oracle or release version changed during reconciliation.

Search preparation error normalization stays in Align direct Rust search validation with the public packed API #65. Broader sustained memory and collector qualification stays in Qualify remaining native metadata collector resource tradeoffs #76. Existing priority/effort/assignment/milestone remain unchanged; no new issue or release commitment is introduced.

Comment generated by kAIle (GPT-6)

Historical proposal — superseded where the supported contract above differs

Outcome

Expose enough low-overhead telemetry to distinguish queue saturation, expensive native work, publication stalls and index growth. Extend the existing status API and native counters; do not create a separate monitoring framework.

Source-inspection baseline: native status and queues and public status API. The inspected implementation already exposes queued commands/bytes, uncommitted mutations, commit opstamp and cumulative queue/execution time. This issue extends that foundation; it is not a claim that the proposed metrics exist today.

First slice

  • Expose queue wait and execution latency distributions for search and writer operations, with operation counts and defined histogram units/buckets. Cumulative nanoseconds alone cannot yield p95/p99.
  • Expose active operations separately from queued operations, queue occupancy versus configured limits, and oldest waiting operation age. Keep ordinary and expensive search queues distinguishable.
  • Count failures, admission rejections, timeouts and cancellations by bounded operation/error category. Define accepted, completed and rejected totals so overload is not hidden by successful-request latency.
  • Expose mutation throughput and logical batch count/bytes/size distributions; distinguish logical batches from transport frames.
  • Separate commit/publication and reader-reload timings where the implementation can measure those stages accurately. Define committed versus reader-visible generation semantics; do not present an opstamp as wall-clock freshness.

Additional diagnostics

  • Expose live-document/segment counts and index disk footprint through supported Tantivy capabilities where available. Specify whether counts describe the writer or a reader snapshot.
  • Collect expensive metadata/directory observations periodically with bounded work and explicit sample age, not a filesystem walk on every scrape.
  • Reuse native merge diagnostics if available; otherwise record the visibility gap rather than reimplementing Tantivy's merge scheduler.
  • Document metric meanings, units, reset/lifecycle behavior, polling cost, snapshot consistency and unavailable values. Relaxed counter reads are observations, not an atomic consistency barrier.
  • Give runnable examples for reading status and exporting metrics without requiring a particular monitoring vendor.

Acceptance and overhead

  • Test counter conservation and timing attribution through success, rejection, timeout, cancellation, close and terminal failure, including concurrent callers and multiple indexes.
  • Bound label cardinality and histogram storage. Never use raw queries, record IDs, user IDs or filesystem paths as metric labels.
  • Keep hot-path measurements allocation-light; make detailed sampling configurable. Measure instrumentation enabled versus disabled across indexing, publication, ordinary search, expensive search and polling. Report throughput, tail latency, CPU and memory cost before choosing defaults.
  • Avoid introducing a global lock or cross-index contention for metrics. Reading telemetry must not queue behind the writer or delay publication.
  • Keep standalone examples executable as tests and update the README/API reference.

Ownership and non-goals

The library owns native operations, queues, files and reader/writer state. Consumers own source-log lag, replication, application readiness, authorization, record hydration and end-to-end request timing.

Process/host CPU, resident memory, page faults and disk latency come from existing process/OS monitoring. Do not invent precise per-index RSS or mmap-residency attribution for shared resources.

No new storage provider, monitoring-server dependency, source-log protocol or alerting system is part of this issue. Do not rebuild Tantivy functionality.

Scheduling

P3 follow-up capability; no release committed. No current open epic specifically owns standalone observability: the search-capabilities epic is scoped to query features, so this issue is deliberately not parented there. Decide histogram API, sampling defaults and acceptable overhead from qualification evidence before implementation.

Comment generated by kAIle (GPT-5)

Lingua principale
Rust
Stelle
3
Fork
0
Merge medio
15h 12m
PR unite (30g)
11

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HarperFast/fulltext

Tutte le issue di HarperFast/fulltext

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.