[BUG] Lucene SimpleText codec is not working - CorruptIndexException
#5,314 opened on Nov 20, 2022
Repository metrics
- Stars
- (8,123 stars)
- PR merge metrics
- (Avg merge 5d 9h) (266 merged PRs in 30d)
Description
Describe the bug Lucene SimpleText codec is not working, leads to CorruptIndexException
To Reproduce
- Start OS 2.0.0 cluster with docker compose (like described on opensearch.org)
- Create an index with
codec: SimpleTextin index settings - Add some documents into this index
- Thats working so far. You can query the documents and cluster status is green
- Shutdown the cluster and restart it and then you will see CorruptIndexException:
opensearch-node1-1 | Caused by: org.apache.lucene.index.CorruptIndexException: verification failed : calculated=1bt1agr stored=skpiqjbymlmy (resource=VerifyingIndexInput(NIOFSIndexInput(path="/usr/share/opensearch/data/nodes/0/indices/gZO2UOXOS5u8D-DKet7fHg/1/index/_0.si")))
opensearch-node1-1 | at org.opensearch.index.store.Store$VerifyingIndexInput.verify(Store.java:1460) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:985) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:912) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.index.store.Store$MetadataSnapshot.<init>(Store.java:825) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.index.store.Store.readMetadataSnapshot(Store.java:480) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.listStoreMetadata(TransportNodesListShardStoreMetadata.java:214) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:155) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:85) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1 | at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) ~[opensearch-2.0.0.jar:2.0.0]
Expected behavior SimpleText codec works like any other lucene codec
Plugins No external plugins, just the one who ships with OS 2.0.0 when using official docker compose file. Just copied the lucene codec jar into the docker image:
COPY lucene-codecs-9.1.0.jar /usr/share/opensearch/lib/lucene-codecs-9.1.0.jar
Screenshots n.a.
Host/Environment (please complete the following information):
- OS: Mac + Docker
- Version: OS 2.0.0
Additional context
I think that the org.opensearch.index.store.Store$VerifyingIndexInput is causing the issue. Looks like it miscalculates the checksum for .si files when Simple Text Codec is used. Thats not happening for the default codec (of course).
It would be great if OS would work with Simple Text Codec because for educational purposes.