[BUG] String(s) terminate after 2000 chars
#6323 aperta il 14 feb 2023
Metriche repository
- Star
- (8123 star)
- Metriche merge PR
- (Merge medio 5g 9h) (266 PR mergiate in 30 g)
Descrizione
Describe the bug
Within some documents, text fields where the length > 2000 chars are 'interrupted' and no longer treated as strings.
When using the reindex API, I received several mapper_parsing_exception failures that indicated string content was being interpreted literally. Such as below:
{
"type" : "mapper_parsing_exception",
"reason" : "failed to parse",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "object field starting or ending with a [.] makes object resolution ambiguous: [\"John is somebody who is going to be a real fighter for the working class ... and I hope very much he joins us in the United States Senate.\"]"
}
}
In OpenSearch Dashboards (OSD), this behaviour can be seen in the syntax highlighting, where the string/text stops being formatted in green at the ~2000 char mark. See screenshot.
To Reproduce Steps to reproduce the behavior:
- Index a document containing a text field where the string length > 2000 chars.
- Potentially, this string may need to be a multiline string.
- Attempt a reindex operation with a painless script that does something with the doc _source.
Plugins
All of 'em -- see AWS OSS Plugins, full output from _cat/plugins is here.
Host/Environment (please complete the following information):
- OpenSearch 2.3
Additional context
- Using one of the broken strings, I calculated the length of the portion that was being treated as a string. Approx. 327 words, 1997 chars.
- I thought that perhaps the "ignore_above" mapping/setting [1] may be to blame, but this does not seem to be the case.
- This may be a Lucene issue as there's a byte limit of 32766, or 32KB. If a field contains UTF-8 text with non-ASCII characters, each char may take up 4 bytes, so the character limit would be 8191 — (32766 / 4 = 8191) [2]
- Again though, the ~2000 character string cut-off point seems to be well-below this number.
- I wrote a painless script to return the length of a string for one of the documents where this broken string issue occurs. The string cutoff (where text is no longer green in OS Dashboards) is also just under 2000 chars, however, the query executes successfully and returns a length of 4396.
- Query uses
params._source['my_broken_field'].toString().length()
- Query uses
[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html [2] https://issues.apache.org/jira/browse/LUCENE-5472