Support versioned per-field analyzers and additional languages

Abierto
  • ¿Algo no está bien?
#52 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
node.js, rust
Área
backend, search

Línea de trabajo

Start with the public API in ts/native.ts and the native implementation in src/engine.rs, then review the README wrapper contract and Tantivy tokenizer capabilities. Define versioned per-field analyzers while preserving english@2, persist options in index identity, and ensure indexing, queries, tracing, and packed packages use matching analysis. Done means the acceptance tests cover Unicode offsets, stop words, synonyms, incompatible reopen behavior, and packed-package behavior, with rebuild requirements documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Outcome

Support versioned per-field analyzers and additional languages.

Current behavior

Verified by source inspection of the public API and native implementation at the assessed revision; no runtime reproduction was performed. The public schema exposes one index-wide english@2 analyzer. Fields containing prose, identifiers, code, or non-English text cannot choose distinct analysis. surfaceTerms already supplies an internal unstemmed companion analyzer for selected operations; the gap is caller-selectable per-field analysis.

Acceptance criteria

  • Add versioned per-field analyzer identities while preserving existing english@2 behavior.
  • Define an initial supported set covering unstemmed text, exact/raw values, additional language stemming, and bounded ngram analysis; document CJK tokenization as requiring separately qualified integrations.
  • Persist analyzer options in index identity and fail closed on incompatible reopen; document rebuild requirements.
  • Use matching analysis in indexing, queries, and tracing; test Unicode offsets, stop words, synonyms, and packed-package behavior.

References and scope

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API.

Tantivy capability. Wrapper contract. This is proposed standalone-library capability with no committed release. Application-specific authorization, source projection, and product API integration are outside this issue.

Deferred analyzer-limit qualification

Follow-up from Add bounded deep-result traversal and qualify autocomplete limits: qualify long-token recall explicitly. Current english@2 omits tokens reaching 40 UTF-8 bytes after processing; this differs from a 40-character bound for multibyte text.

  • Define supported prose/identifier/language behavior at ASCII and multibyte boundaries. Explain how omitted query terms affect Boolean matching, not only indexing.
  • Preserve english@2 compatibility; introduce a versioned identity and rebuild for changed tokenizer/filter semantics. Do not silently raise the constant in place.

Comment generated by kAIle (GPT-6)

Delivery sequence and Harper integration

Proposed sequence for implementing this issue; the full acceptance criteria above remain open:

  1. Establish per-field versioned analyzer selection, identity compatibility and capability validation. Deliver unstemmed text and exact/raw profiles first; preserve existing english@2 output and long-token behavior.
  2. Add selected language profiles with explicit supported-language and offset/recall fixtures.
  3. Add bounded ngrams after defining index growth, query behavior and interactions with prefix/fuzzy modes. CJK remains separately qualified.

For each increment, use matching indexing, query and tracing analysis, reject unsupported combinations explicitly, and test actual packed native artifacts. Final profile names/options and costs require design qualification; these phases are not a claim that all profiles are turnkey or committed to a release.

Harper schema exposure, persistence/introspection, generation rebuild and deployed query tests are owned by Expose versioned full-text analyzer selection through Harper schema declarations. The wrapper retains native analyzer implementation and standalone qualification. Existing priority, effort, parent, type and milestone are preserved.

Comment generated by kAIle (GPT-6)

Lenguaje dominante
Rust
Estrellas
3
Forks
0
Merge medio
15 h 12 min
PR fusionados (30 d)
11

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de HarperFast/fulltext

Todos los issues de HarperFast/fulltext

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.