elastic/elasticsearch

Fail index analyzer that contains a graph token filter

Open

#24,396 创建于 2017年4月28日

在 GitHub 查看
 (3 评论) (0 反应) (0 负责人)Java (76,700 star) (25,882 fork)batch import
:Search Relevance/Analysis>enhancementTeam:Search Relevancehelp wanted

描述

Currently it is possible to set a synonym_graph or a word_delimiter_graph token filter in an analyzer that is used at index time. Though these filters can produce side-paths that will break the positions in the index and make phrase query matching impossible on the field. The flatten_graph token filter is supposed to handle this situation but it can only flatten the graph which is also a lossy operation. So whether the user adds a flatten_graph filter at the end of the analyzer or not the positions of the terms in the index will not be accurate. Instead we could try to detect these situation and fail the mapping if a graph filter is used in an index analyzer. This would allow us to remove the flatten_graph filter and also help users to not shoot themselves in the foot. Here is an hopefully exhaustive list of token filters that should be impacted by this:

  • synonym_graph_filter
  • word_delimiter_graph_filter
  • shingles (only when output_unigram:true or min_size < max_size)
  • cjk (only when output_unigram:true)
  • ngram tokenizer when min_gram < max_gram
  • common_gram
  • kuromoji_tokenizer when (nbest_cost or nbest_example > 1).

贡献者指南