elastic/elasticsearch

Fail index analyzer that contains a graph token filter

Open

#24,396 opened on 2017年4月28日

GitHub で見る
 (3 comments) (0 reactions) (0 assignees)Java (76,700 stars) (25,882 forks)batch import
:Search Relevance/Analysis>enhancementTeam:Search Relevancehelp wanted

説明

Currently it is possible to set a synonym_graph or a word_delimiter_graph token filter in an analyzer that is used at index time. Though these filters can produce side-paths that will break the positions in the index and make phrase query matching impossible on the field. The flatten_graph token filter is supposed to handle this situation but it can only flatten the graph which is also a lossy operation. So whether the user adds a flatten_graph filter at the end of the analyzer or not the positions of the terms in the index will not be accurate. Instead we could try to detect these situation and fail the mapping if a graph filter is used in an index analyzer. This would allow us to remove the flatten_graph filter and also help users to not shoot themselves in the foot. Here is an hopefully exhaustive list of token filters that should be impacted by this:

  • synonym_graph_filter
  • word_delimiter_graph_filter
  • shingles (only when output_unigram:true or min_size < max_size)
  • cjk (only when output_unigram:true)
  • ngram tokenizer when min_gram < max_gram
  • common_gram
  • kuromoji_tokenizer when (nbest_cost or nbest_example > 1).

コントリビューターガイド

技術スタック
java
領域
backendsearch
Issue 種別
feature
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
3
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
1-2 days
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
stale
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
mostly clear
前提条件
Understanding of Elasticsearch analyzersBasic Java knowledge
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
35
調査方針
The issue proposes to detect graph token filters in index analyzers and fail the mapping. Currently, the flatten graph filter is used as a workaround. The codebase likely validates analyzer chains in the analysis module. Start by examining the token filter registration and analyzer building logic, e.g., in `AnalysisRegistry` or `AnalyzerProvider`. Add a validation step that checks for synonym graph filter, word delimiter graph filter, shingles (when output unigram:true or min size < max size), cjk (when output unigram:true), ngram tokenizer (when min gram < max gram), common gram, and kuromoji tokenizer (when nbest cost or nbest example > 1). When detected, throw an error. Also consider removing the flatten graph filter. The affected filters are listed in the issue.