Lucene 9 miswrites some long strings with unpaired surrogates

The fix is meant to live in the Lucene index backend, where strings are handed to Lucene for storage and analysis, and where LuceneIndex.restore reads documents back. Start by reading the reproduction and the mentioned LuceneIndex.restore path, then check how the open pull request #5112 already changes those call sites. Done means documents with unpaired surrogates store, read back and reindex without errors, and textContains finds words at the end of such values.

On master, since the upgrade to Lucene 9.12.3, a Lucene mixed index can write a document which it can't read back, or reads back with a string cut short. It happens to a string longer than 1,024 UTF-16 units which has an unpaired high surrogate right before a surrogate pair, such as an emoji, where that surrogate is the last unit of one of the windows of 1,023 units in which Lucene writes the string: the 1,023rd unit, the 2,046th, and so on, one unit later for each window before it which ended in a high surrogate. The cause is a bug in Lucene's ByteBuffersDataOutput.writeLongString. JanusGraph 1.1.x isn't affected: its Lucene 8.11 writes such strings right.

Reproduction, in the Gremlin Console (master dd4f0f845):

graph = JanusGraphFactory.open('conf/janusgraph-berkeleyje-lucene.properties')
mgmt = graph.openManagement()
text = mgmt.makePropertyKey('text').dataType(String.class).make()
name = mgmt.makePropertyKey('name').dataType(String.class).make()
mgmt.buildIndex('byText', Vertex.class).addKey(text).addKey(name, Mapping.STRING.asParameter()).buildMixedIndex('search')
mgmt.commit()

g = graph.traversal()
value = 'x' * 1022 + '\uD800\uD83D\uDE00 tail words'   // an unpaired high surrogate as the 1,023rd unit, then U+1F600
v = g.addV().property('text', value).next()
g.tx().commit()
g.V().has('text', textContains('words')).toList()      // [v]
g.V(v).property('name', 'first').iterate()
g.tx().commit()
g.V().has('text', textContains('words')).toList()      // expected [v], actual []

• Cut short: where the string is the last stored field of its document, as here, the document reads back with the string's last 2 bytes cut off, and its surrogates as U+FFFD. The update reads the document back and indexes the string again from what Lucene stored, ...tail wor, so textContains('words') no longer finds the vertex. Queries which read the string whole, such as an equality on a key mapped as STRING, miss it the same way.
• Unreadable: where another stored field follows the string, the document can't be read at all: a query which finds it fails with JanusGraphException: Could not call index, caused by a NullPointerException in Lucene's DocumentStoredFieldVisitor, an ArrayIndexOutOfBoundsException or an AssertionError, such as "Unknown type flag: 7", which Lucene throws with assertions off too. Every later index mutation of the element fails too, after its storage commit ("Error while committing index mutations"), as it reads the document first, and so do the updates of the documents after it in the same store and transaction; on master, those before it stay uncommitted in the IndexWriter until the store's next commit. Which of the two happens to an element given both text and name in one transaction depends on the order of its index updates: of six such vertices, five became unreadable and one was cut short.
• Reindexing doesn't repair such a document: LuceneIndex.restore reads the old document before it replaces it, and fails on it with a PermanentBackendException caused by the NullPointerException, which isn't reattempted. So a reindex doesn't write the documents after it in its batch of up to schema.reindex.mixed-index-batch-size (1,000) either, cut-short ones included. Removing the element deletes it; for an index which may hold such documents, deleting the directory of its store, index.[X].directory/<index name>, while no instance runs, and reindexing replaces them all.

With Lucene alone (lucene-core 9.12.3):

String text = "x".repeat(1022) + "\uD800\uD83D\uDE00 tail words"; // an unpaired high surrogate, then U+1F600
try (Directory dir = new ByteBuffersDirectory();
     IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig())) {
    Document doc = new Document();
    doc.add(new StoredField("text", text));
    doc.add(new StoredField("id", "doc1"));
    writer.addDocument(doc);
    writer.commit();
    try (DirectoryReader reader = DirectoryReader.open(dir)) {
        reader.storedFields().document(0); // NullPointerException
    }
}

Without the id field, the document reads back, and the string ends in \uFFFD\uFFFD\uFFFD tail wor: 1,034 units of 1,036. With 1,025 units, the string ending right after the pair, the read fails with AssertionError: Unknown type flag: 7. lucene-core 10.5.1 and 10.5.2 fail just the same, on Java 21, and 8.11.3 reads the string back whole.

Cause: ByteBuffersDataOutput.writeString writes a string of more than 1,024 units through writeLongString. That declares the string's UTF-8 length, from UnicodeUtil.calcUTF16toUTF8Length, and encodes the string in windows of at most 1,023 units, lengthening a window which ends in a high surrogate by one unit, so as not to split a pair:

int step = Math.min(end - i, MAX_CHARS_PER_WINDOW - 1);
if (i + step < end && Character.isHighSurrogate(s.charAt(i + step - 1))) {
  step++;
}
int upTo = UnicodeUtil.UTF16toUTF8(s, i, step, buf);

Where that high surrogate is unpaired and a surrogate pair follows it, the extra unit is the high half of the pair, and its low half starts the next window. UTF16toUTF8 writes U+FFFD for each half, 3 bytes each, where calcUTF16toUTF8Length counted 4 for the pair, so the field holds 2 bytes more than it declares. A reader of the document lands 2 bytes early: on the next field's header, or, past the last field, it just reads the string short. The windows came with apache/lucene#12455 (July 2023). On Lucene's main branch these lines are the same, though writeString encodes a string directly where it fits in the current block of its buffer, and takes writeLongString only where it doesn't.

A second problem, which JanusGraph 1.1.x has too: Lucene's standard tokenizer, which keys mapped as DEFAULT, TEXT and TEXTSTRING use by default, drops the words which it reads last along with an unpaired high surrogate at the end of the text: its zzRefill, finding a read which ends in a high surrogate, reads one unit more for the low half, and takes the end of the text for the end of input, without the units which it just read. So:

• A value which ends in an unpaired high surrogate is indexed without those words, all of them in a short value, until a later write of its document analyzes the string as Lucene stored it, with U+FFFD: textContains("words") doesn't find "Tail Words\uD800" until its element is updated. Of "word " 100 times followed by \uD800, the tokenizer gives 51 words; lucene-core 8.11.3 and 9.12.3 do the same.
• A query value which ends so gives no words: textContains("words\uD800") adds no condition, and finds every element of the index, and textContainsPrefix("word\uD800") looks for words starting with word\uFFFD and finds none.

Proposed fix: hand Lucene each string with U+FFFD in place of its unpaired surrogates, to store and to analyze, and analyze the string values of queries so too. That's how Lucene stores such a string in every other case, so the stored bytes don't change, Lucene never sees an unpaired surrogate, and the analyzer gets the same string on the first write as on later ones.