opensearch-project/OpenSearch

[BUG] Lucene SimpleText codec is not working - CorruptIndexException

Open

#5,314 opened on Nov 20, 2022

View on GitHub
 (3 comments) (1 reaction) (1 assignee)Java (1,505 forks)batch import
Indexingbuggood first issuelucene

Repository metrics

Stars
 (8,123 stars)
PR merge metrics
 (Avg merge 5d 9h) (266 merged PRs in 30d)

Description

Describe the bug Lucene SimpleText codec is not working, leads to CorruptIndexException

To Reproduce

  • Start OS 2.0.0 cluster with docker compose (like described on opensearch.org)
  • Create an index with codec: SimpleText in index settings
  • Add some documents into this index
  • Thats working so far. You can query the documents and cluster status is green
  • Shutdown the cluster and restart it and then you will see CorruptIndexException:
opensearch-node1-1  | Caused by: org.apache.lucene.index.CorruptIndexException: verification failed : calculated=1bt1agr stored=skpiqjbymlmy (resource=VerifyingIndexInput(NIOFSIndexInput(path="/usr/share/opensearch/data/nodes/0/indices/gZO2UOXOS5u8D-DKet7fHg/1/index/_0.si")))
opensearch-node1-1  |     at org.opensearch.index.store.Store$VerifyingIndexInput.verify(Store.java:1460) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:985) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:912) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store$MetadataSnapshot.<init>(Store.java:825) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store.readMetadataSnapshot(Store.java:480) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.listStoreMetadata(TransportNodesListShardStoreMetadata.java:214) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:155) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:85) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) ~[opensearch-2.0.0.jar:2.0.0]

Expected behavior SimpleText codec works like any other lucene codec

Plugins No external plugins, just the one who ships with OS 2.0.0 when using official docker compose file. Just copied the lucene codec jar into the docker image:

COPY lucene-codecs-9.1.0.jar /usr/share/opensearch/lib/lucene-codecs-9.1.0.jar

Screenshots n.a.

Host/Environment (please complete the following information):

  • OS: Mac + Docker
  • Version: OS 2.0.0

Additional context I think that the org.opensearch.index.store.Store$VerifyingIndexInput is causing the issue. Looks like it miscalculates the checksum for .si files when Simple Text Codec is used. Thats not happening for the default codec (of course).

It would be great if OS would work with Simple Text Codec because for educational purposes.

Contributor guide