Support versioned per-field analyzers and additional languages

オープン
  • 何か問題がありますか？
#52 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
node.js, rust
領域
backend, search

調査の方向性

Start with the public API in ts/native.ts and the native implementation in src/engine.rs, then review the README wrapper contract and Tantivy tokenizer capabilities. Define versioned per-field analyzers while preserving english@2, persist options in index identity, and ensure indexing, queries, tracing, and packed packages use matching analysis. Done means the acceptance tests cover Unicode offsets, stop words, synonyms, incompatible reopen behavior, and packed-package behavior, with rebuild requirements documented.

索引モデルが issue の本文から書いたものです。

説明

Outcome

Support versioned per-field analyzers and additional languages.

Current behavior

Verified by source inspection of the public API and native implementation at the assessed revision; no runtime reproduction was performed. The public schema exposes one index-wide english@2 analyzer. Fields containing prose, identifiers, code, or non-English text cannot choose distinct analysis. surfaceTerms already supplies an internal unstemmed companion analyzer for selected operations; the gap is caller-selectable per-field analysis.

Acceptance criteria

  • Add versioned per-field analyzer identities while preserving existing english@2 behavior.
  • Define an initial supported set covering unstemmed text, exact/raw values, additional language stemming, and bounded ngram analysis; document CJK tokenization as requiring separately qualified integrations.
  • Persist analyzer options in index identity and fail closed on incompatible reopen; document rebuild requirements.
  • Use matching analysis in indexing, queries, and tracing; test Unicode offsets, stop words, synonyms, and packed-package behavior.

References and scope

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API.

Tantivy capability. Wrapper contract. This is proposed standalone-library capability with no committed release. Application-specific authorization, source projection, and product API integration are outside this issue.

Deferred analyzer-limit qualification

Follow-up from Add bounded deep-result traversal and qualify autocomplete limits: qualify long-token recall explicitly. Current english@2 omits tokens reaching 40 UTF-8 bytes after processing; this differs from a 40-character bound for multibyte text.

  • Define supported prose/identifier/language behavior at ASCII and multibyte boundaries. Explain how omitted query terms affect Boolean matching, not only indexing.
  • Preserve english@2 compatibility; introduce a versioned identity and rebuild for changed tokenizer/filter semantics. Do not silently raise the constant in place.

Comment generated by kAIle (GPT-6)

Delivery sequence and Harper integration

Proposed sequence for implementing this issue; the full acceptance criteria above remain open:

  1. Establish per-field versioned analyzer selection, identity compatibility and capability validation. Deliver unstemmed text and exact/raw profiles first; preserve existing english@2 output and long-token behavior.
  2. Add selected language profiles with explicit supported-language and offset/recall fixtures.
  3. Add bounded ngrams after defining index growth, query behavior and interactions with prefix/fuzzy modes. CJK remains separately qualified.

For each increment, use matching indexing, query and tracing analysis, reject unsupported combinations explicitly, and test actual packed native artifacts. Final profile names/options and costs require design qualification; these phases are not a claim that all profiles are turnkey or committed to a release.

Harper schema exposure, persistence/introspection, generation rebuild and deployed query tests are owned by Expose versioned full-text analyzer selection through Harper schema declarations. The wrapper retains native analyzer implementation and standalone qualification. Existing priority, effort, parent, type and milestone are preserved.

Comment generated by kAIle (GPT-6)

主要言語
Rust
スター
3
フォーク
0
平均マージ
15時間 12分
マージ済み PR（30日）
11

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

HarperFast/fulltext のほかの issue

HarperFast/fulltext の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。