opensearch-project/OpenSearch

Opensearch is not working in IPv6 cluster "handshake failed. unexpected remote cluster"

Open

#3 043 ouverte le 10 mars 2022

Voir sur GitHub
 (7 commentaires) (0 réactions) (0 assignés)Java (1 505 forks)batch import
bugdistributed frameworkhelp wanted

Métriques du dépôt

Stars
 (8 123 stars)
Métriques de merge PR
 (Merge moyen 5j 9h) (266 PRs mergées en 30 j)

Description

Hi All,

https://github.com/opensearch-project/helm-charts/issues/119 I tried deploying opensearch helm chart in my IPv4 k8s cluster and I am getting the below responses in pod logs and I was able to see the cluster health status.

[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin https://opensearch-cluster-master:9200/_cluster/health?pretty=true
{
  "cluster_name" : "opensearch-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "active_primary_shards" : 1,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

This is my k8s nodes details where its working fine and without any issues

[root@k8master1 ganesh]# kubectl get nodes -o wide
NAME          STATUS     ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
10.69.2.150   Ready      master   131d   v1.17.9   10.69.2.150   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.151   Ready      master   131d   v1.17.9   10.69.2.151   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.152   Ready      master   131d   v1.17.9   10.69.2.152   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.156   Ready      worker   131d   v1.17.9   10.69.2.156   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.157   Ready      worker   131d   v1.17.9   10.69.2.157   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.158   Ready      worker   131d   v1.17.9   10.69.2.158   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.159   NotReady   worker   131d   v1.17.9   10.69.2.159   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://Unknown
10.69.2.160   NotReady   worker   131d   v1.17.9   10.69.2.160   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.161   Ready      worker   131d   v1.17.9   10.69.2.161   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.162   Ready      worker   131d   v1.17.9   10.69.2.162   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.163   Ready      worker   131d   v1.17.9   10.69.2.163   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.164   Ready      worker   131d   v1.17.9   10.69.2.164   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11

and when I tested the same in other k8s environment with IPv6 cluster I am getting SSL error and below is the IPv6 k8s node details,

[LRB_346_PCAA@k8s-rmp-master-0 kafka]$ sudo kubectl get nodes -o wide
NAME                            STATUS   ROLES    AGE    VERSION   INTERNAL-IP                        EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-rmp-master-0                Ready    master   503d   v1.20.5   fcff:69:70:0:faf2:1eff:fe89:b8f0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-master-1                Ready    master   503d   v1.20.5   fcff:69:70:0:faf2:1eff:fe89:cce0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-master-2                Ready    master   503d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:a9a0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-0                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:d210   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-1                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:af70   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-10               Ready    <none>   370d   v1.20.5   fcff:69:70:0:faf2:1eff:fe65:2f68   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-11               Ready    <none>   341d   v1.20.5   fcff:69:70:0:faf2:1eff:feb1:e0c0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-12               Ready    <none>   342d   v1.20.5   fcff:69:70:0:faf2:1eff:fe98:79c4   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-13               Ready    <none>   301d   v1.20.5   fcff:69:70::90                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-14               Ready    <none>   301d   v1.20.5   fcff:69:70::91                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-15               Ready    <none>   301d   v1.20.5   fcff:69:70::92                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-16               Ready    <none>   104d   v1.20.5   fcff:69:70::93                     <none>        CentOS Linux 7 (Core)   3.10.0-1160.21.1.el7.x86_64   robin://19.3.6
k8s-rmp-worker-17               Ready    <none>   71d    v1.20.5   fcff:69:70::94                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-18               Ready    <none>   69d    v1.20.5   fcff:69:70::95                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-2                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:a320   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-3                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:b2d0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-4                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:bb40   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-5                Ready    <none>   441d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:aab0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-6                Ready    <none>   467d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:b450   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-7                Ready    <none>   456d   v1.20.5   fcff:69:70:0:faf2:1eff:fe8a:cf0    <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-8                Ready    <none>   456d   v1.20.5   fcff:69:70:0:faf2:1eff:fe89:9b70   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-9                Ready    <none>   369d   v1.20.5   fcff:69:70:0:faf2:1eff:fe65:2dc4   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6

and below is the SSL error I am getting in the IPv6 k8s environment pod logs,

[2021-09-29T05:20:46,769][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] [connectToRemoteMasterNode[**[fd74:ca9b:3a09:868c:172:18:0:4f52]:9300**]] completed handshake with [{​​​​​opensearch-cluster-master-0}​​​​​{​​​​​iRiB2GoKTtiXE5uSCsb-pw}​​​​​{​​​​​_sJJFLKBToGMK5fK00ugHQ}​​​​​{​​​​​127.0.0.1}​​​​​{​​​​​127.0.0.1:9300}​​​​​{​​​​​dimr}​​​​​] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {​​​​​opensearch-cluster-master-1}​​​​​{​​​​​56Z5v568SMuOVmLk4QMhKg}​​​​​{​​​​​2VL7vr5rSjyoFdZyg5Pruw}​​​​​{​​​​​127.0.0.1}​​​​​{​​​​​127.0.0.1:9300}​​​​​{​​​​​dimr}​​​​​
master-1 IPv6 is resolving but followup connection with master-0 is failing bcs of the Ipv4 address (127.0.0.1) and this is the same behaviour for master-2 when its trying to talk to master-0

also I tried netstat inside the master-0 pod below is the response and its trying to connect IPv4

[opensearch@opensearch-cluster-master-0 ~]$ /tmp/netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::9600                 :::*                    LISTEN      17/java
tcp6       0      0 :::9200                 :::*                    LISTEN      9/java
tcp6       0      0 :::9650                 :::*                    LISTEN      17/java
tcp6       0      0 :::9300                 :::*                    LISTEN      9/java
tcp6       0      0 127.0.0.1:42684         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:35322         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:60987         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:43450         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:40486         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:58494         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:34366         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:40803         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:44544         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 fd74:ca9b:3a09:86:33982 fd74:ca9b:3a09:868:9300 TIME_WAIT   -

Note:- The same helm chart was working fine in IPv4 k8s cluster.

Below is the logs details I am seeing in pods,

opensearch-cluster-master-0

Node[[fd74:ca9b:3a09:868c:172:18:0:4320]:9300]] completed handshake with [{opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:21:35,583][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:482e]:9300]] completed handshake with [{opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-2][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}

opensearch-cluster-master-1

[2022-03-07T13:23:01,443][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:44a6]:9300]] completed handshake with [{opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:23:02,427][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:482e]:9300]] completed handshake with [{opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-2][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

opensearch-cluster-master-2

[2022-03-07T13:24:02,257][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4320]:9300]] completed handshake with [{opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:03,260][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:44a6]:9300]] completed handshake with [{opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:03,267][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4320]:9300]] completed handshake with [{opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:04,201][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-2] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:202) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:188) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:76) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:554) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:04,255][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:44a6]:9300]] completed handshake with [{opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}

Kindly help me to resolve this issue and please correct me if I am doing anything wrong.

Regards, Ganeshbabu R

Guide contributeur

Direction de recherche
Examinez la configuration d'OpenSearch pour les paramètres 'network.host', 'transport.host' et 'discovery.seed hosts' afin de garantir qu'ils utilisent des adresses IPv6 dans les clusters IPv6. Vérifiez les modèles de chart Helm pour les variables d'environnement ou les remplacements de configuration qui pourraient forcer IPv4. Testez un correctif en définissant 'network.host' sur ' site ' ou en utilisant explicitement l'adresse IPv6 du nœud. Consultez également l'issue #119 dans le dépôt helm charts pour le contexte associé.
Stack technique
java
Domaine
backendapi
Type d'issue
Bug
DifficultéDifficulté estimée pour un nouveau contributeur, de 1 pour un très petit changement à 5 pour un travail expert.
3
Temps estiméFourchette de temps approximative pour investiguer, implémenter, tester et préparer une pull request.
1-2 jours
Statut d'activitéDisponibilité apparente de l'issue : fraîche, active, ancienne, bloquée ou en attente d'un mainteneur.
Active
ClartéClarté avec laquelle l'issue explique le changement attendu, les critères d'acceptation et la prochaine étape.
Plutôt claire
Prérequis
KubernetesOpenSearch
Accessibilité débutantScore de 1 à 100 estimant l'accessibilité de cette issue pour un premier contributeur.
60