Opensearch elasticsearch cluster health couldn't able to get after node restarted

Hi Team,

We have done the opendsearch installation in 3 nodes. At the first time we could able to get cluster health details in all 3 nodes like below.

[elastic@es2 opensearch]$ curl -k -u admin:admin -XGET https://es2:9200/_cluster/health?pretty { "cluster_name" : "opensearch-elasticsearch", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "discovered_master" : true, "active_primary_shards" : 1, "active_shards" : 3, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } [elastic@es2 opensearch]$ date Fri Jul 16 17:11:09 IST 2021 [elastic@es opensearch]$

After restarted the es1 host then we got below error.

[elastic@es1 opensearch]$ curl -k -u admin:admin -XGET https://es1:9200/_cluster/health?pretty { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "Unexpected exception cluster:monitor/health" } ], "type" : "security_exception", "reason" : "Unexpected exception cluster:monitor/health" }, "status" : 500 } [elastic@es1 opensearch]$ date Fri Jul 16 17:13:26 IST 2021 [elastic@es1 opensearch]$

We got the below error in logs.

[2021-07-16T17:13:17,436][ERROR][o.o.s.f.SecurityFilter ] [es1] Unexpected exception java.lang.ExceptionInInitializerError java.lang.ExceptionInInitializerError: null

If es2 host restarted then we have similar issue, same issue for es3 host too.

If all 3 hosts restarted then we could able to got cluster health only in 1 node(we con't say exactly purticular node). We couldn't able get cluster health in all 3 nodes until unless removed every thing in all 3 nodes then we have to freshly installation.

We have followed the below steps to install:

1. After untar the package "opensearch-1.0.0-linux-x64.tar.gz" then added the below lines to opensearch.yml

cluster.name: opensearch-elasticsearch node.name: ${HOSTNAME} path.data: /var/lib/scylla/elastic/opensearch/data path.logs: /var/lib/scylla/elastic/opensearch/logs network.host: x.x.x.x http.port: 9200 transport.tcp.port: 9300 discovery.seed_hosts: ["x.x.x.x:9300","x.x.x.x:9300","x.x.x.x:9300"] cluster.initial_master_nodes: ["x.x.x.x","x.x.x.x","x.x.x.x"]

Same config for other 2 nodes too except network.host

2. Started the scipt "opensearch-tar-install.sh"

Please help to come out from this issue.

Thank you.