opensearch-project/OpenSearch

Opensearch is not working in IPv6 cluster "handshake failed. unexpected remote cluster"

Open

#3,043 建立於 2022年3月10日

在 GitHub 查看
 (7 留言) (0 反應) (0 負責人)Java (1,505 fork)batch import
bugdistributed frameworkhelp wanted

倉庫指標

Star
 (8,123 star)
PR 合併指標
 (平均合併 5天 9小時) (30 天內合併 266 個 PR)

描述

Hi All,

https://github.com/opensearch-project/helm-charts/issues/119 I tried deploying opensearch helm chart in my IPv4 k8s cluster and I am getting the below responses in pod logs and I was able to see the cluster health status.

[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin https://opensearch-cluster-master:9200/_cluster/health?pretty=true
{
  "cluster_name" : "opensearch-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "active_primary_shards" : 1,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

This is my k8s nodes details where its working fine and without any issues

[root@k8master1 ganesh]# kubectl get nodes -o wide
NAME          STATUS     ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
10.69.2.150   Ready      master   131d   v1.17.9   10.69.2.150   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.151   Ready      master   131d   v1.17.9   10.69.2.151   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.152   Ready      master   131d   v1.17.9   10.69.2.152   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.156   Ready      worker   131d   v1.17.9   10.69.2.156   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.157   Ready      worker   131d   v1.17.9   10.69.2.157   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.158   Ready      worker   131d   v1.17.9   10.69.2.158   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.159   NotReady   worker   131d   v1.17.9   10.69.2.159   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://Unknown
10.69.2.160   NotReady   worker   131d   v1.17.9   10.69.2.160   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.161   Ready      worker   131d   v1.17.9   10.69.2.161   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.162   Ready      worker   131d   v1.17.9   10.69.2.162   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.163   Ready      worker   131d   v1.17.9   10.69.2.163   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.164   Ready      worker   131d   v1.17.9   10.69.2.164   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11

and when I tested the same in other k8s environment with IPv6 cluster I am getting SSL error and below is the IPv6 k8s node details,

[LRB_346_PCAA@k8s-rmp-master-0 kafka]$ sudo kubectl get nodes -o wide
NAME                            STATUS   ROLES    AGE    VERSION   INTERNAL-IP                        EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-rmp-master-0                Ready    master   503d   v1.20.5   fcff:69:70:0:faf2:1eff:fe89:b8f0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-master-1                Ready    master   503d   v1.20.5   fcff:69:70:0:faf2:1eff:fe89:cce0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-master-2                Ready    master   503d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:a9a0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-0                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:d210   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-1                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:af70   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-10               Ready    <none>   370d   v1.20.5   fcff:69:70:0:faf2:1eff:fe65:2f68   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-11               Ready    <none>   341d   v1.20.5   fcff:69:70:0:faf2:1eff:feb1:e0c0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-12               Ready    <none>   342d   v1.20.5   fcff:69:70:0:faf2:1eff:fe98:79c4   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-13               Ready    <none>   301d   v1.20.5   fcff:69:70::90                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-14               Ready    <none>   301d   v1.20.5   fcff:69:70::91                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-15               Ready    <none>   301d   v1.20.5   fcff:69:70::92                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-16               Ready    <none>   104d   v1.20.5   fcff:69:70::93                     <none>        CentOS Linux 7 (Core)   3.10.0-1160.21.1.el7.x86_64   robin://19.3.6
k8s-rmp-worker-17               Ready    <none>   71d    v1.20.5   fcff:69:70::94                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-18               Ready    <none>   69d    v1.20.5   fcff:69:70::95                     <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-2                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:a320   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-3                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:b2d0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-4                Ready    <none>   494d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:bb40   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-5                Ready    <none>   441d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:aab0   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-6                Ready    <none>   467d   v1.20.5   fcff:69:70:0:faf2:1eff:fe84:b450   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-7                Ready    <none>   456d   v1.20.5   fcff:69:70:0:faf2:1eff:fe8a:cf0    <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-8                Ready    <none>   456d   v1.20.5   fcff:69:70:0:faf2:1eff:fe89:9b70   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6
k8s-rmp-worker-9                Ready    <none>   369d   v1.20.5   fcff:69:70:0:faf2:1eff:fe65:2dc4   <none>        CentOS Linux 7 (Core)   3.10.0-1062.el7.x86_64        robin://19.3.6

and below is the SSL error I am getting in the IPv6 k8s environment pod logs,

[2021-09-29T05:20:46,769][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] [connectToRemoteMasterNode[**[fd74:ca9b:3a09:868c:172:18:0:4f52]:9300**]] completed handshake with [{​​​​​opensearch-cluster-master-0}​​​​​{​​​​​iRiB2GoKTtiXE5uSCsb-pw}​​​​​{​​​​​_sJJFLKBToGMK5fK00ugHQ}​​​​​{​​​​​127.0.0.1}​​​​​{​​​​​127.0.0.1:9300}​​​​​{​​​​​dimr}​​​​​] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {​​​​​opensearch-cluster-master-1}​​​​​{​​​​​56Z5v568SMuOVmLk4QMhKg}​​​​​{​​​​​2VL7vr5rSjyoFdZyg5Pruw}​​​​​{​​​​​127.0.0.1}​​​​​{​​​​​127.0.0.1:9300}​​​​​{​​​​​dimr}​​​​​
master-1 IPv6 is resolving but followup connection with master-0 is failing bcs of the Ipv4 address (127.0.0.1) and this is the same behaviour for master-2 when its trying to talk to master-0

also I tried netstat inside the master-0 pod below is the response and its trying to connect IPv4

[opensearch@opensearch-cluster-master-0 ~]$ /tmp/netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::9600                 :::*                    LISTEN      17/java
tcp6       0      0 :::9200                 :::*                    LISTEN      9/java
tcp6       0      0 :::9650                 :::*                    LISTEN      17/java
tcp6       0      0 :::9300                 :::*                    LISTEN      9/java
tcp6       0      0 127.0.0.1:42684         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:35322         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:60987         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:43450         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:40486         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:58494         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:34366         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:40803         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 127.0.0.1:44544         127.0.0.1:9300          TIME_WAIT   -
tcp6       0      0 fd74:ca9b:3a09:86:33982 fd74:ca9b:3a09:868:9300 TIME_WAIT   -

Note:- The same helm chart was working fine in IPv4 k8s cluster.

Below is the logs details I am seeing in pods,

opensearch-cluster-master-0

Node[[fd74:ca9b:3a09:868c:172:18:0:4320]:9300]] completed handshake with [{opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:21:35,583][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:482e]:9300]] completed handshake with [{opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-2][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}

opensearch-cluster-master-1

[2022-03-07T13:23:01,443][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:44a6]:9300]] completed handshake with [{opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:23:02,427][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:482e]:9300]] completed handshake with [{opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-2][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

opensearch-cluster-master-2

[2022-03-07T13:24:02,257][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4320]:9300]] completed handshake with [{opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:03,260][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:44a6]:9300]] completed handshake with [{opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:03,267][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4320]:9300]] completed handshake with [{opensearch-cluster-master-1}{HChcGYj4QJyRVhmvmjFKaw}{NytJh4T9RKqPgiNU3z7UEg}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:439) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:172) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:555) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.TransportService$5.onResponse(TransportService.java:528) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:67) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:319) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:313) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) [opensearch-1.2.3.jar:1.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:04,201][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-2] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:202) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:188) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:76) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:554) [opensearch-1.2.3.jar:1.2.3]
        at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.2.3.0.jar:1.2.3.0]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-07T13:24:04,255][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:44a6]:9300]] completed handshake with [{opensearch-cluster-master-0}{UiaYM8hfRsqiYyEh5h7FxQ}{qfwEvselRBS6mZTwiI0gYQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {opensearch-cluster-master-2}{OwfMlWSxRliGrxRiOSbvdg}{SCRtvfflTyektNk44H30rw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}

Kindly help me to resolve this issue and please correct me if I am doing anything wrong.

Regards, Ganeshbabu R

貢獻者指南

研究方向
調查 OpenSearch 配置中 'network.host'、'transport.host' 和 'discovery.seed hosts' 的設定，確保在 IPv6 叢集中使用 IPv6 位址。檢查 Helm chart 範本中可能強制使用 IPv4 的環境變數或配置覆寫。透過將 'network.host' 設定為 ' site ' 或明確使用節點的 IPv6 位址來測試修復。同時，查看 helm charts 儲存庫中的 issue #119 以取得相關上下文。
技術棧
java
領域
backendapi
議題類型
錯誤
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-2 天
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
活躍
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
大致清晰
前置要求
KubernetesOpenSearch
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
60