opensearch-project/OpenSearch

[BUG] Lucene SimpleText codec is not working - CorruptIndexException

Open

#5314 aperta il 20 nov 2022

Vedi su GitHub
 (3 commenti) (1 reazione) (1 assegnatario)Java (1505 fork)batch import
Indexingbuggood first issuelucene

Metriche repository

Star
 (8123 star)
Metriche merge PR
 (Merge medio 5g 9h) (266 PR mergiate in 30 g)

Descrizione

Describe the bug Lucene SimpleText codec is not working, leads to CorruptIndexException

To Reproduce

  • Start OS 2.0.0 cluster with docker compose (like described on opensearch.org)
  • Create an index with codec: SimpleText in index settings
  • Add some documents into this index
  • Thats working so far. You can query the documents and cluster status is green
  • Shutdown the cluster and restart it and then you will see CorruptIndexException:
opensearch-node1-1  | Caused by: org.apache.lucene.index.CorruptIndexException: verification failed : calculated=1bt1agr stored=skpiqjbymlmy (resource=VerifyingIndexInput(NIOFSIndexInput(path="/usr/share/opensearch/data/nodes/0/indices/gZO2UOXOS5u8D-DKet7fHg/1/index/_0.si")))
opensearch-node1-1  |     at org.opensearch.index.store.Store$VerifyingIndexInput.verify(Store.java:1460) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:985) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:912) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store$MetadataSnapshot.<init>(Store.java:825) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.index.store.Store.readMetadataSnapshot(Store.java:480) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.listStoreMetadata(TransportNodesListShardStoreMetadata.java:214) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:155) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.indices.store.TransportNodesListShardStoreMetadata.nodeOperation(TransportNodesListShardStoreMetadata.java:85) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) ~[opensearch-2.0.0.jar:2.0.0]
opensearch-node1-1  |     at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) ~[opensearch-2.0.0.jar:2.0.0]

Expected behavior SimpleText codec works like any other lucene codec

Plugins No external plugins, just the one who ships with OS 2.0.0 when using official docker compose file. Just copied the lucene codec jar into the docker image:

COPY lucene-codecs-9.1.0.jar /usr/share/opensearch/lib/lucene-codecs-9.1.0.jar

Screenshots n.a.

Host/Environment (please complete the following information):

  • OS: Mac + Docker
  • Version: OS 2.0.0

Additional context I think that the org.opensearch.index.store.Store$VerifyingIndexInput is causing the issue. Looks like it miscalculates the checksum for .si files when Simple Text Codec is used. Thats not happening for the default codec (of course).

It would be great if OS would work with Simple Text Codec because for educational purposes.

Guida contributor