elastic/elasticsearch

Fail index analyzer that contains a graph token filter

Open

#24396 opened on Apr 28, 2017

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Java (76,700 stars) (25,882 forks)batch import
:Search Relevance/Analysis>enhancementTeam:Search Relevancehelp wanted

Description

Currently it is possible to set a synonym_graph or a word_delimiter_graph token filter in an analyzer that is used at index time. Though these filters can produce side-paths that will break the positions in the index and make phrase query matching impossible on the field. The flatten_graph token filter is supposed to handle this situation but it can only flatten the graph which is also a lossy operation. So whether the user adds a flatten_graph filter at the end of the analyzer or not the positions of the terms in the index will not be accurate. Instead we could try to detect these situation and fail the mapping if a graph filter is used in an index analyzer. This would allow us to remove the flatten_graph filter and also help users to not shoot themselves in the foot. Here is an hopefully exhaustive list of token filters that should be impacted by this:

  • synonym_graph_filter
  • word_delimiter_graph_filter
  • shingles (only when output_unigram:true or min_size < max_size)
  • cjk (only when output_unigram:true)
  • ngram tokenizer when min_gram < max_gram
  • common_gram
  • kuromoji_tokenizer when (nbest_cost or nbest_example > 1).

Contributor guide

Tech stack
java
Domain
backendsearch
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Understanding of Elasticsearch analyzersBasic Java knowledge
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
The issue proposes to detect graph token filters in index analyzers and fail the mapping. Currently, the flatten graph filter is used as a workaround. The codebase likely validates analyzer chains in the analysis module. Start by examining the token filter registration and analyzer building logic, e.g., in `AnalysisRegistry` or `AnalyzerProvider`. Add a validation step that checks for synonym graph filter, word delimiter graph filter, shingles (when output unigram:true or min size < max size), cjk (when output unigram:true), ngram tokenizer (when min gram < max gram), common gram, and kuromoji tokenizer (when nbest cost or nbest example > 1). When detected, throw an error. Also consider removing the flatten graph filter. The affected filters are listed in the issue.