opensearch-project/OpenSearch

[BUG] String(s) terminate after 2000 chars

Open

#6,323 opened on Feb 14, 2023

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Java (1,505 forks)batch import
Indexingbuggood first issuelucene

Repository metrics

Stars
 (8,123 stars)
PR merge metrics
 (Avg merge 5d 9h) (266 merged PRs in 30d)

Description

Describe the bug Within some documents, text fields where the length > 2000 chars are 'interrupted' and no longer treated as strings.

When using the reindex API, I received several mapper_parsing_exception failures that indicated string content was being interpreted literally. Such as below:

{
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse",
    "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "object field starting or ending with a [.] makes object resolution ambiguous: [\"John is somebody who is going to be a real fighter for the working class ... and I hope very much he joins us in the United States Senate.\"]"
    }
}

In OpenSearch Dashboards (OSD), this behaviour can be seen in the syntax highlighting, where the string/text stops being formatted in green at the ~2000 char mark. See screenshot.

To Reproduce Steps to reproduce the behavior:

  1. Index a document containing a text field where the string length > 2000 chars.
    • Potentially, this string may need to be a multiline string.
  2. Attempt a reindex operation with a painless script that does something with the doc _source.

Plugins All of 'em -- see AWS OSS Plugins, full output from _cat/plugins is here.

Host/Environment (please complete the following information):

  • OpenSearch 2.3

Additional context

  1. Using one of the broken strings, I calculated the length of the portion that was being treated as a string. Approx. 327 words, 1997 chars.
    • I thought that perhaps the "ignore_above" mapping/setting [1] may be to blame, but this does not seem to be the case.
  2. This may be a Lucene issue as there's a byte limit of 32766, or 32KB. If a field contains UTF-8 text with non-ASCII characters, each char may take up 4 bytes, so the character limit would be 8191 — (32766 / 4 = 8191) [2]
    • Again though, the ~2000 character string cut-off point seems to be well-below this number.
  3. I wrote a painless script to return the length of a string for one of the documents where this broken string issue occurs. The string cutoff (where text is no longer green in OS Dashboards) is also just under 2000 chars, however, the query executes successfully and returns a length of 4396.
    • Query uses params._source['my_broken_field'].toString().length()

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html [2] https://issues.apache.org/jira/browse/LUCENE-5472

Contributor guide