Support versioned per-field analyzers and additional languages

Aperta
  • Qualcosa non va?
#52 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
node.js, rust
Ambito
backend, search

Direzione di ricerca

Start with the public API in ts/native.ts and the native implementation in src/engine.rs, then review the README wrapper contract and Tantivy tokenizer capabilities. Define versioned per-field analyzers while preserving english@2, persist options in index identity, and ensure indexing, queries, tracing, and packed packages use matching analysis. Done means the acceptance tests cover Unicode offsets, stop words, synonyms, incompatible reopen behavior, and packed-package behavior, with rebuild requirements documented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Outcome

Support versioned per-field analyzers and additional languages.

Current behavior

Verified by source inspection of the public API and native implementation at the assessed revision; no runtime reproduction was performed. The public schema exposes one index-wide english@2 analyzer. Fields containing prose, identifiers, code, or non-English text cannot choose distinct analysis. surfaceTerms already supplies an internal unstemmed companion analyzer for selected operations; the gap is caller-selectable per-field analysis.

Acceptance criteria

  • Add versioned per-field analyzer identities while preserving existing english@2 behavior.
  • Define an initial supported set covering unstemmed text, exact/raw values, additional language stemming, and bounded ngram analysis; document CJK tokenization as requiring separately qualified integrations.
  • Persist analyzer options in index identity and fail closed on incompatible reopen; document rebuild requirements.
  • Use matching analysis in indexing, queries, and tracing; test Unicode offsets, stop words, synonyms, and packed-package behavior.

References and scope

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API.

Tantivy capability. Wrapper contract. This is proposed standalone-library capability with no committed release. Application-specific authorization, source projection, and product API integration are outside this issue.

Deferred analyzer-limit qualification

Follow-up from Add bounded deep-result traversal and qualify autocomplete limits: qualify long-token recall explicitly. Current english@2 omits tokens reaching 40 UTF-8 bytes after processing; this differs from a 40-character bound for multibyte text.

  • Define supported prose/identifier/language behavior at ASCII and multibyte boundaries. Explain how omitted query terms affect Boolean matching, not only indexing.
  • Preserve english@2 compatibility; introduce a versioned identity and rebuild for changed tokenizer/filter semantics. Do not silently raise the constant in place.

Comment generated by kAIle (GPT-6)

Delivery sequence and Harper integration

Proposed sequence for implementing this issue; the full acceptance criteria above remain open:

  1. Establish per-field versioned analyzer selection, identity compatibility and capability validation. Deliver unstemmed text and exact/raw profiles first; preserve existing english@2 output and long-token behavior.
  2. Add selected language profiles with explicit supported-language and offset/recall fixtures.
  3. Add bounded ngrams after defining index growth, query behavior and interactions with prefix/fuzzy modes. CJK remains separately qualified.

For each increment, use matching indexing, query and tracing analysis, reject unsupported combinations explicitly, and test actual packed native artifacts. Final profile names/options and costs require design qualification; these phases are not a claim that all profiles are turnkey or committed to a release.

Harper schema exposure, persistence/introspection, generation rebuild and deployed query tests are owned by Expose versioned full-text analyzer selection through Harper schema declarations. The wrapper retains native analyzer implementation and standalone qualification. Existing priority, effort, parent, type and milestone are preserved.

Comment generated by kAIle (GPT-6)

Lingua principale
Rust
Stelle
3
Fork
0
Merge medio
15h 12m
PR unite (30g)
11

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HarperFast/fulltext

Tutte le issue di HarperFast/fulltext

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.